Jul 13 18:49:38.356652 ip-10-0-139-134 systemd[1]: Starting Kubernetes Kubelet...
Jul 13 18:49:38.813609 ip-10-0-139-134 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 18:49:38.813609 ip-10-0-139-134 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 13 18:49:38.813609 ip-10-0-139-134 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 18:49:38.813609 ip-10-0-139-134 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 13 18:49:38.813609 ip-10-0-139-134 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 18:49:38.814335 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.813695    2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 13 18:49:38.817959 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.817937    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:49:38.817959 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.817957    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:49:38.817959 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.817961    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:49:38.818052 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.817964    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:49:38.818052 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.817967    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:49:38.818052 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.817970    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:49:38.818052 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.817973    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:49:38.818052 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.817976    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:49:38.818052 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.817982    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:49:38.818052 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.817986    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:49:38.818052 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.817989    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:49:38.818052 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.817999    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:49:38.818052 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818002    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:49:38.818052 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818005    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:49:38.818052 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818008    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:49:38.818052 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818010    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:49:38.818052 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818013    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:49:38.818052 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818017    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:49:38.818052 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818020    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:49:38.818052 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818024    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:49:38.818052 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818026    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:49:38.818052 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818029    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:49:38.818521 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818032    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:49:38.818521 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818035    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:49:38.818521 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818037    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:49:38.818521 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818040    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:49:38.818521 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818042    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:49:38.818521 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818045    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:49:38.818521 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818047    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:49:38.818521 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818050    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:49:38.818521 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818052    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:49:38.818521 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818055    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:49:38.818521 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818058    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:49:38.818521 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818061    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:49:38.818521 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818066    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:49:38.818521 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818068    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:49:38.818521 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818072    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:49:38.818521 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818076    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:49:38.818521 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818079    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:49:38.818521 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818081    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:49:38.818521 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818084    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:49:38.818521 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818087    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:49:38.819002 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818089    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:49:38.819002 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818091    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:49:38.819002 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818094    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:49:38.819002 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818096    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:49:38.819002 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818099    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:49:38.819002 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818101    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:49:38.819002 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818103    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:49:38.819002 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818106    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:49:38.819002 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818108    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:49:38.819002 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818110    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:49:38.819002 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818113    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:49:38.819002 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818115    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:49:38.819002 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818118    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:49:38.819002 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818121    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:49:38.819002 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818124    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:49:38.819002 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818126    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:49:38.819002 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818129    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:49:38.819002 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818131    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:49:38.819002 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818134    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:49:38.819002 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818137    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:49:38.819558 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818139    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:49:38.819558 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818142    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:49:38.819558 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818144    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:49:38.819558 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818146    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:49:38.819558 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818149    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:49:38.819558 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818151    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:49:38.819558 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818153    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:49:38.819558 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818156    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:49:38.819558 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818159    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:49:38.819558 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818161    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:49:38.819558 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818164    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:49:38.819558 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818166    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:49:38.819558 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818169    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:49:38.819558 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818171    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:49:38.819558 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818174    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:49:38.819558 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818176    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:49:38.819558 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818178    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:49:38.819558 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818181    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:49:38.819558 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818183    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:49:38.820310 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818187    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:49:38.820310 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818189    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:49:38.820310 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818192    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:49:38.820310 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818194    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:49:38.820310 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818196    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:49:38.820310 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818199    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:49:38.820310 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818202    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:49:38.820310 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818205    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:49:38.820310 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818208    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:49:38.820310 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818210    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:49:38.820310 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818212    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:49:38.820310 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818215    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:49:38.820310 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818217    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:49:38.820310 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818219    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:49:38.820310 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818222    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:49:38.820310 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818225    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:49:38.820310 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818227    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:49:38.820310 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818230    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:49:38.820310 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818948    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:49:38.820310 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818955    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:49:38.820994 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818958    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:49:38.820994 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818961    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:49:38.820994 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818964    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:49:38.820994 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818966    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:49:38.820994 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818969    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:49:38.820994 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818972    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:49:38.820994 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818974    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:49:38.820994 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818977    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:49:38.820994 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818979    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:49:38.820994 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818982    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:49:38.820994 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818984    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:49:38.820994 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818987    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:49:38.820994 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818989    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:49:38.820994 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818992    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:49:38.820994 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818994    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:49:38.820994 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818997    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:49:38.820994 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.818999    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:49:38.820994 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819002    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:49:38.820994 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819005    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:49:38.820994 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819008    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:49:38.821622 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819012    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:49:38.821622 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819015    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:49:38.821622 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819017    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:49:38.821622 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819020    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:49:38.821622 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819022    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:49:38.821622 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819025    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:49:38.821622 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819028    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:49:38.821622 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819030    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:49:38.821622 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819033    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:49:38.821622 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819035    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:49:38.821622 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819038    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:49:38.821622 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819040    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:49:38.821622 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819042    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:49:38.821622 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819045    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:49:38.821622 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819048    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:49:38.821622 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819050    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:49:38.821622 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819053    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:49:38.821622 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819055    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:49:38.821622 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819058    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:49:38.821622 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819060    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:49:38.822156 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819063    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:49:38.822156 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819065    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:49:38.822156 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819067    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:49:38.822156 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819070    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:49:38.822156 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819072    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:49:38.822156 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819075    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:49:38.822156 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819077    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:49:38.822156 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819079    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:49:38.822156 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819083    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:49:38.822156 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819086    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:49:38.822156 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819091    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:49:38.822156 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819094    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:49:38.822156 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819097    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:49:38.822156 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819099    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:49:38.822156 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819102    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:49:38.822156 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819106    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:49:38.822156 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819108    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:49:38.822156 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819111    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:49:38.822156 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819113    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:49:38.822156 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819116    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:49:38.822671 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819118    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:49:38.822671 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819121    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:49:38.822671 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819124    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:49:38.822671 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819126    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:49:38.822671 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819128    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:49:38.822671 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819131    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:49:38.822671 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819133    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:49:38.822671 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819138    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:49:38.822671 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819142    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:49:38.822671 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819145    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:49:38.822671 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819148    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:49:38.822671 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819151    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:49:38.822671 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819154    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:49:38.822671 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819157    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:49:38.822671 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819160    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:49:38.822671 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819162    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:49:38.822671 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819165    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:49:38.822671 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819168    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:49:38.822671 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819170    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:49:38.823136 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819172    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:49:38.823136 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819174    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:49:38.823136 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819177    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:49:38.823136 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819180    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:49:38.823136 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819182    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:49:38.823136 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819185    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:49:38.823136 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819188    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:49:38.823136 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819191    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:49:38.823136 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819194    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:49:38.823136 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819196    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:49:38.823136 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819199    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:49:38.823136 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819202    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:49:38.823136 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819205    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:49:38.823136 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819208    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:49:38.823136 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819211    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:49:38.823136 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819214    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:49:38.823136 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819217    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:49:38.823136 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.819219    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:49:38.823136 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820737    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:49:38.823136 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820750    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:49:38.823650 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820754    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:49:38.823650 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820757    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:49:38.823650 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820761    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:49:38.823650 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820764    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:49:38.823650 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820767    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:49:38.823650 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820771    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:49:38.823650 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820773    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:49:38.823650 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820776    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:49:38.823650 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820785    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:49:38.823650 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820788    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:49:38.823650 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820921    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:49:38.823650 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820925    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:49:38.823650 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820928    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:49:38.823650 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820932    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:49:38.823650 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820937    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:49:38.823650 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820940    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:49:38.823650 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820943    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:49:38.823650 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820945    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:49:38.823650 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820948    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:49:38.824207 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820951    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:49:38.824207 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820953    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:49:38.824207 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820956    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:49:38.824207 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820958    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:49:38.824207 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820961    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:49:38.824207 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820963    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:49:38.824207 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820966    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:49:38.824207 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820968    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:49:38.824207 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820971    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:49:38.824207 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820973    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:49:38.824207 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820975    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:49:38.824207 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820979    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:49:38.824207 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820982    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:49:38.824207 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820986    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:49:38.824207 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820991    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:49:38.824207 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820994    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:49:38.824207 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820997    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:49:38.824207 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.820999    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:49:38.824207 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821002    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:49:38.824207 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821005    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:49:38.824705 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821008    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:49:38.824705 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821010    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:49:38.824705 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821014    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:49:38.824705 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821016    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:49:38.824705 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821019    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:49:38.824705 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821021    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:49:38.824705 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821024    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:49:38.824705 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821028    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:49:38.824705 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821030    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:49:38.824705 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821033    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:49:38.824705 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821035    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:49:38.824705 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821038    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:49:38.824705 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821040    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:49:38.824705 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821042    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:49:38.824705 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821045    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:49:38.824705 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821047    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:49:38.824705 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821049    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:49:38.824705 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821052    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:49:38.824705 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821054    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:49:38.824705 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821057    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:49:38.825180 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821059    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:49:38.825180 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821062    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:49:38.825180 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821064    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:49:38.825180 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821068    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:49:38.825180 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821071    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:49:38.825180 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821074    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:49:38.825180 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821077    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:49:38.825180 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821079    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:49:38.825180 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821082    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:49:38.825180 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821084    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:49:38.825180 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821087    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:49:38.825180 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821090    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:49:38.825180 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821092    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:49:38.825180 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821095    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:49:38.825180 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821097    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:49:38.825180 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821100    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:49:38.825180 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821103    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:49:38.825180 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821105    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:49:38.825180 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821108    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:49:38.825180 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821110    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:49:38.825699 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821113    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:49:38.825699 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821115    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:49:38.825699 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821118    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:49:38.825699 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821123    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:49:38.825699 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821126    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:49:38.825699 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821129    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:49:38.825699 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821132    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:49:38.825699 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821135    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:49:38.825699 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821138    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:49:38.825699 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821140    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:49:38.825699 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821143    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:49:38.825699 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821145    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:49:38.825699 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821148    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:49:38.825699 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821150    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:49:38.825699 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821153    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:49:38.825699 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821156    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:49:38.825699 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821158    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:49:38.825699 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821161    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:49:38.825699 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821238    2569 flags.go:64] FLAG: --address="0.0.0.0"
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821247    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821254    2569 flags.go:64] FLAG: --anonymous-auth="true"
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821258    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821263    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821266    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821271    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821275    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821279    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821282    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821300    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821304    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821307    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821310    2569 flags.go:64] FLAG: --cgroup-root=""
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821312    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821315    2569 flags.go:64] FLAG: --client-ca-file=""
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821318    2569 flags.go:64] FLAG: --cloud-provider="external"
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821322    2569 flags.go:64] FLAG: --cluster-dns="[]"
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821327    2569 flags.go:64] FLAG: --cluster-domain=""
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821329    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821332    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821336    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821339    2569 flags.go:64] FLAG: --container-log-max-files="5"
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821343    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 13 18:49:38.826152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821346    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821349    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821352    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821355    2569 flags.go:64] FLAG: --contention-profiling="false"
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821358    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821361    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821364    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821367    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821371    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821374    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821377    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821380    2569 flags.go:64] FLAG: --enable-load-reader="false"
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821383    2569 flags.go:64] FLAG: --enable-server="true"
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821386    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821390    2569 flags.go:64] FLAG: --event-burst="100"
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821393    2569 flags.go:64] FLAG: --event-qps="50"
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821396    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821399    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821402    2569 flags.go:64] FLAG: --eviction-hard=""
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821405    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821408    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821411    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821414    2569 flags.go:64] FLAG: --eviction-soft=""
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821417    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821419    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 13 18:49:38.826731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821422    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821425    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821428    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821431    2569 flags.go:64] FLAG: --fail-swap-on="true"
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821434    2569 flags.go:64] FLAG: --feature-gates=""
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821438    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821441    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821444    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821447    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821450    2569 flags.go:64] FLAG: --healthz-port="10248"
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821453    2569 flags.go:64] FLAG: --help="false"
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821456    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-139-134.ec2.internal"
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821460    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821463    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821465    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821468    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821472    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821474    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821478    2569 flags.go:64] FLAG: --image-service-endpoint=""
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821481    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821484    2569 flags.go:64] FLAG: --kube-api-burst="100"
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821486    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821489    2569 flags.go:64] FLAG: --kube-api-qps="50"
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821492    2569 flags.go:64] FLAG: --kube-reserved=""
Jul 13 18:49:38.827445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821495    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821497    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821500    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821503    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821506    2569 flags.go:64] FLAG: --lock-file=""
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821508    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821511    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821514    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821524    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821527    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821532    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821535    2569 flags.go:64] FLAG: --logging-format="text"
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821538    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821541    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821543    2569 flags.go:64] FLAG: --manifest-url=""
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821546    2569 flags.go:64] FLAG: --manifest-url-header=""
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821551    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821554    2569 flags.go:64] FLAG: --max-open-files="1000000"
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821558    2569 flags.go:64] FLAG: --max-pods="110"
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821561    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821564    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821567    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821570    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821573    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821576    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 13 18:49:38.828040 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821579    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821586    2569 flags.go:64] FLAG: --node-status-max-images="50"
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821589    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821592    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821595    2569 flags.go:64] FLAG: --pod-cidr=""
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821597    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821602    2569 flags.go:64] FLAG: --pod-manifest-path=""
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821605    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821608    2569 flags.go:64] FLAG: --pods-per-core="0"
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821610    2569 flags.go:64] FLAG: --port="10250"
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821614    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821617    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e0cb488c0119563f"
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821620    2569 flags.go:64] FLAG: --qos-reserved=""
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821623    2569 flags.go:64] FLAG: --read-only-port="10255"
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821626    2569 flags.go:64] FLAG: --register-node="true"
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821629    2569 flags.go:64] FLAG: --register-with-taints=""
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821632    2569 flags.go:64] FLAG: --registry-burst="10"
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821635    2569 flags.go:64] FLAG: --registry-qps="5"
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821638    2569 flags.go:64] FLAG: --reserved-cpus=""
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821641    2569 flags.go:64] FLAG: --reserved-memory=""
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821648    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821652    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821655    2569 flags.go:64] FLAG: --rotate-certificates="false"
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821657    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821660    2569 flags.go:64] FLAG: --runonce="false"
Jul 13 18:49:38.828673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821663    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821666    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821670    2569 flags.go:64] FLAG: --seccomp-default="false"
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821673    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821676    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821679    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821681    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821684    2569 flags.go:64] FLAG: --storage-driver-password="root"
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821687    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821689    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821692    2569 flags.go:64] FLAG: --storage-driver-user="root"
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821695    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821698    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821701    2569 flags.go:64] FLAG: --system-cgroups=""
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821703    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821709    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821712    2569 flags.go:64] FLAG: --tls-cert-file=""
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821715    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821719    2569 flags.go:64] FLAG: --tls-min-version=""
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821722    2569 flags.go:64] FLAG: --tls-private-key-file=""
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821724    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821727    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821730    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821733    2569 flags.go:64] FLAG: --v="2"
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821737    2569 flags.go:64] FLAG: --version="false"
Jul 13 18:49:38.829308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821741    2569 flags.go:64] FLAG: --vmodule=""
Jul 13 18:49:38.829917 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821745    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 13 18:49:38.829917 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.821750    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 13 18:49:38.829917 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821866    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:49:38.829917 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821869    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:49:38.829917 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821872    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:49:38.829917 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821875    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:49:38.829917 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821877    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:49:38.829917 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821880    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:49:38.829917 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821885    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:49:38.829917 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821887    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:49:38.829917 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821890    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:49:38.829917 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821892    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:49:38.829917 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821895    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:49:38.829917 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821898    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:49:38.829917 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821900    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:49:38.829917 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821903    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:49:38.829917 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821906    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:49:38.829917 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821908    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:49:38.829917 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821911    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:49:38.829917 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821913    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:49:38.830430 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821916    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:49:38.830430 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821918    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:49:38.830430 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821920    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:49:38.830430 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821923    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:49:38.830430 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821925    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:49:38.830430 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821928    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:49:38.830430 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821931    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:49:38.830430 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821933    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:49:38.830430 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821936    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:49:38.830430 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821938    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:49:38.830430 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821941    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:49:38.830430 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821945    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:49:38.830430 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821949    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:49:38.830430 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821954    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:49:38.830430 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821956    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:49:38.830430 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821959    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:49:38.830430 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821962    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:49:38.830430 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821965    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:49:38.830430 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821968    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:49:38.830898 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821970    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:49:38.830898 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821974    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:49:38.830898 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821977    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:49:38.830898 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821979    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:49:38.830898 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821981    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:49:38.830898 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821984    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:49:38.830898 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821986    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:49:38.830898 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821989    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:49:38.830898 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821991    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:49:38.830898 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821994    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:49:38.830898 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821997    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:49:38.830898 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.821999    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:49:38.830898 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822002    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:49:38.830898 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822004    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:49:38.830898 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822006    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:49:38.830898 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822009    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:49:38.830898 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822011    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:49:38.830898 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822014    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:49:38.830898 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822016    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:49:38.830898 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822019    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:49:38.831523 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822022    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:49:38.831523 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822024    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:49:38.831523 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822027    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:49:38.831523 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822030    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:49:38.831523 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822034    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:49:38.831523 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822037    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:49:38.831523 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822041    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:49:38.831523 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822044    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:49:38.831523 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822046    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:49:38.831523 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822049    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:49:38.831523 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822052    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:49:38.831523 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822055    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:49:38.831523 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822059    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:49:38.831523 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822063    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:49:38.831523 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822066    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:49:38.831523 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822068    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:49:38.831523 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822071    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:49:38.831523 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822074    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:49:38.831523 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822077    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:49:38.831986 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822079    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:49:38.831986 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822082    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:49:38.831986 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822084    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:49:38.831986 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822086    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:49:38.831986 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822089    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:49:38.831986 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822091    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:49:38.831986 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822099    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:49:38.831986 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822102    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:49:38.831986 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822104    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:49:38.831986 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822107    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:49:38.831986 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822109    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:49:38.831986 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822112    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:49:38.831986 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822114    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:49:38.831986 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822117    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:49:38.831986 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822119    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:49:38.831986 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822122    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:49:38.831986 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822125    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:49:38.831986 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822127    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:49:38.831986 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822129    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:49:38.831986 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822133    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:49:38.832488 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822135    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:49:38.832488 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822138    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:49:38.832488 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.822141    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:49:38.832488 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.822953    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 18:49:38.832488 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.830473    2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 13 18:49:38.832488 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.830496    2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 13 18:49:38.832488 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.830511    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 13 18:49:38.832488 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.830516    2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 13 18:49:38.832488 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830575    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:49:38.832488 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830580    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:49:38.832488 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830583    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:49:38.832488 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830586    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:49:38.832488 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830589    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:49:38.832488 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830592    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:49:38.832838 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830596    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:49:38.832838 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830600    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:49:38.832838 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830603    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:49:38.832838 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830606    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:49:38.832838 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830609    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:49:38.832838 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830612    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:49:38.832838 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830615    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:49:38.832838 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830617    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:49:38.832838 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830620    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:49:38.832838 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830624    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:49:38.832838 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830628    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:49:38.832838 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830631    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:49:38.832838 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830633    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:49:38.832838 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830636    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:49:38.832838 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830639    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:49:38.832838 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830641    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:49:38.832838 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830644    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:49:38.832838 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830646    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:49:38.832838 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830649    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:49:38.833303 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830652    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:49:38.833303 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830654    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:49:38.833303 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830657    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:49:38.833303 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830659    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:49:38.833303 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830663    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:49:38.833303 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830666    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:49:38.833303 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830669    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:49:38.833303 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830672    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:49:38.833303 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830674    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:49:38.833303 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830677    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:49:38.833303 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830679    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:49:38.833303 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830681    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:49:38.833303 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830685    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:49:38.833303 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830687    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:49:38.833303 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830690    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:49:38.833303 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830692    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:49:38.833303 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830695    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:49:38.833303 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830697    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:49:38.833303 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830700    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:49:38.833303 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830702    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:49:38.833788 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830705    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:49:38.833788 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830707    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:49:38.833788 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830710    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:49:38.833788 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830712    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:49:38.833788 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830715    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:49:38.833788 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830717    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:49:38.833788 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830720    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:49:38.833788 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830722    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:49:38.833788 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830725    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:49:38.833788 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830727    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:49:38.833788 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830730    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:49:38.833788 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830732    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:49:38.833788 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830734    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:49:38.833788 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830737    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:49:38.833788 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830740    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:49:38.833788 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830742    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:49:38.833788 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830745    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:49:38.833788 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830749    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:49:38.833788 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830751    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:49:38.834244 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830754    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:49:38.834244 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830756    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:49:38.834244 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830759    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:49:38.834244 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830761    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:49:38.834244 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830763    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:49:38.834244 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830766    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:49:38.834244 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830768    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:49:38.834244 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830771    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:49:38.834244 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830773    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:49:38.834244 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830776    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:49:38.834244 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830778    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:49:38.834244 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830781    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:49:38.834244 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830783    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:49:38.834244 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830785    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:49:38.834244 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830788    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:49:38.834244 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830791    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:49:38.834244 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830793    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:49:38.834244 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830796    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:49:38.834244 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830798    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:49:38.834244 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830800    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:49:38.834756 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830803    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:49:38.834756 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830805    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:49:38.834756 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830808    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:49:38.834756 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830810    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:49:38.834756 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830812    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:49:38.834756 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830815    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:49:38.834756 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830817    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:49:38.834756 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830820    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:49:38.834756 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830822    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:49:38.834756 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830826    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:49:38.834756 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830830    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:49:38.834756 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830834    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:49:38.834756 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830837    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:49:38.834756 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830839    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:49:38.834756 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830842    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:49:38.835124 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.830847    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 18:49:38.835124 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830968    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:49:38.835124 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830972    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:49:38.835124 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830975    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:49:38.835124 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830978    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:49:38.835124 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830981    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:49:38.835124 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830983    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:49:38.835124 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830986    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:49:38.835124 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830988    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:49:38.835124 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830991    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:49:38.835124 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830993    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:49:38.835124 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830995    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:49:38.835124 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.830998    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:49:38.835124 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831000    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:49:38.835505 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831003    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:49:38.835505 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831005    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:49:38.835505 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831008    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:49:38.835505 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831010    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:49:38.835505 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831013    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:49:38.835505 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831015    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:49:38.835505 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831017    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:49:38.835505 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831020    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:49:38.835505 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831022    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:49:38.835505 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831025    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:49:38.835505 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831027    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:49:38.835505 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831030    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:49:38.835505 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831033    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:49:38.835505 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831035    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:49:38.835505 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831038    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:49:38.835505 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831041    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:49:38.835505 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831043    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:49:38.835505 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831046    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:49:38.835505 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831048    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:49:38.835505 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831051    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:49:38.835990 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831053    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:49:38.835990 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831056    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:49:38.835990 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831058    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:49:38.835990 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831061    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:49:38.835990 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831063    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:49:38.835990 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831067    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:49:38.835990 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831070    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:49:38.835990 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831073    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:49:38.835990 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831076    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:49:38.835990 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831079    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:49:38.835990 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831081    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:49:38.835990 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831084    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:49:38.835990 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831086    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:49:38.835990 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831089    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:49:38.835990 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831091    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:49:38.835990 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831094    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:49:38.835990 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831096    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:49:38.835990 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831098    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:49:38.835990 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831101    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:49:38.836472 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831103    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:49:38.836472 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831106    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:49:38.836472 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831109    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:49:38.836472 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831111    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:49:38.836472 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831114    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:49:38.836472 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831117    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:49:38.836472 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831119    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:49:38.836472 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831122    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:49:38.836472 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831124    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:49:38.836472 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831127    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:49:38.836472 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831130    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:49:38.836472 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831132    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:49:38.836472 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831135    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:49:38.836472 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831137    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:49:38.836472 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831140    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:49:38.836472 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831142    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:49:38.836472 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831145    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:49:38.836472 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831147    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:49:38.836472 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831149    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:49:38.836472 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831152    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:49:38.836952 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831154    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:49:38.836952 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831157    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:49:38.836952 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831159    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:49:38.836952 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831161    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:49:38.836952 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831164    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:49:38.836952 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831166    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:49:38.836952 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831169    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:49:38.836952 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831171    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:49:38.836952 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831173    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:49:38.836952 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831176    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:49:38.836952 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831178    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:49:38.836952 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831181    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:49:38.836952 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831183    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:49:38.836952 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831185    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:49:38.836952 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831188    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:49:38.836952 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831190    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:49:38.836952 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831193    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:49:38.836952 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831196    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:49:38.836952 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831199    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:49:38.836952 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831201    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:49:38.837464 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831204    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:49:38.837464 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831207    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:49:38.837464 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831211    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:49:38.837464 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831214    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:49:38.837464 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831216    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:49:38.837464 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831219    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:49:38.837464 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:38.831222    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:49:38.837464 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.831227    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 18:49:38.837464 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.831395    2569 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 13 18:49:38.837464 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.833927    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 13 18:49:38.837464 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.835682    2569 server.go:1019] "Starting client certificate rotation"
Jul 13 18:49:38.837464 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.835793    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 18:49:38.837464 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.835871    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 18:49:38.838658 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.838638    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 18:49:38.842693 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.842668    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 18:49:38.858584 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.858560    2569 log.go:25] "Validated CRI v1 runtime API"
Jul 13 18:49:38.860992 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.860978    2569 log.go:25] "Validated CRI v1 image API"
Jul 13 18:49:38.861787 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.861774    2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 13 18:49:38.864458 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.864432    2569 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8ef4d3e4-23ed-4646-b449-29f04d28fdcf:/dev/nvme0n1p4 c2561256-8daf-46e8-9a09-7f5d64e5b087:/dev/nvme0n1p3]
Jul 13 18:49:38.864518 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.864457    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 13 18:49:38.867651 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.867562    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 18:49:38.870274 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.870160    2569 manager.go:216] Machine: {Timestamp:2026-07-13 18:49:38.869620172 +0000 UTC m=+0.383419328 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3066322 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24921ff56566bb76e3d3463f43b02c SystemUUID:ec24921f-f565-66bb-76e3-d3463f43b02c BootID:9b195bf0-2656-4a53-9936-64a4e18acb6c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8b:13:33:31:bb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8b:13:33:31:bb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:b6:5e:85:88:2e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 13 18:49:38.870274 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.870267    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 13 18:49:38.870405 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.870364    2569 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 13 18:49:38.873337 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.873305    2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 13 18:49:38.873478 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.873339    2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-134.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 13 18:49:38.874279 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.874267    2569 topology_manager.go:138] "Creating topology manager with none policy"
Jul 13 18:49:38.874339 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.874281    2569 container_manager_linux.go:309] "Creating device plugin manager"
Jul 13 18:49:38.874339 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.874306    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 18:49:38.874339 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.874319    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 18:49:38.876321 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.876309    2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 13 18:49:38.881085 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.881071    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 18:49:38.881208 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.881199    2569 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 13 18:49:38.881240 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.881232    2569 kubelet.go:486] "Attempting to sync node with API server"
Jul 13 18:49:38.881273 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.881241    2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 13 18:49:38.881273 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.881256    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 13 18:49:38.881273 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.881265    2569 kubelet.go:398] "Adding apiserver pod source"
Jul 13 18:49:38.881371 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.881276    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 13 18:49:38.881672 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.881660    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 18:49:38.881710 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.881679    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 18:49:38.883995 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.883979    2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 13 18:49:38.886242 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.886226    2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 13 18:49:38.886336 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.886265    2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 13 18:49:38.886492 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.886479    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 13 18:49:38.886556 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.886498    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 13 18:49:38.886556 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.886507    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 13 18:49:38.886556 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.886516    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 13 18:49:38.886556 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.886529    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 13 18:49:38.886556 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.886537    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 13 18:49:38.887139 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.887130    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 13 18:49:38.887185 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.887142    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 13 18:49:38.887185 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.887152    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 13 18:49:38.887185 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.887161    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 13 18:49:38.887185 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.887181    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 13 18:49:38.887372 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.887194    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 13 18:49:38.888932 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.888920    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 13 18:49:38.888981 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.888935    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 13 18:49:38.892235 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.892218    2569 server.go:1268] "Started kubelet"
Jul 13 18:49:38.892386 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.892328    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 13 18:49:38.892449 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.892413    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 13 18:49:38.892884 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.892857    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 13 18:49:38.893025 ip-10-0-139-134 systemd[1]: Started Kubernetes Kubelet.
Jul 13 18:49:38.893283 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.893250    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-134.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 13 18:49:38.894002 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:38.893978    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-134.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 13 18:49:38.894074 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:38.893981    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 13 18:49:38.894179 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.894164    2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 13 18:49:38.895547 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.895532    2569 server.go:310] "Adding debug handlers to kubelet server"
Jul 13 18:49:38.898417 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:38.898396    2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 13 18:49:38.900453 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:38.899309    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-134.ec2.internal.18c1edf87b53cab3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-134.ec2.internal,UID:ip-10-0-139-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-134.ec2.internal,},FirstTimestamp:2026-07-13 18:49:38.892188339 +0000 UTC m=+0.405987496,LastTimestamp:2026-07-13 18:49:38.892188339 +0000 UTC m=+0.405987496,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-134.ec2.internal,}"
Jul 13 18:49:38.901730 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.901710    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 13 18:49:38.901831 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.901713    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 13 18:49:38.901831 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:38.901820    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-134.ec2.internal\" not found"
Jul 13 18:49:38.902191 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.902127    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vx8l9"
Jul 13 18:49:38.902481 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.902452    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 13 18:49:38.902585 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.902453    2569 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 13 18:49:38.902585 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.902499    2569 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 13 18:49:38.902799 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.902623    2569 reconstruct.go:97] "Volume reconstruction finished"
Jul 13 18:49:38.902799 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.902634    2569 reconciler.go:29] "Reconciler: start to sync state"
Jul 13 18:49:38.904523 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.904500    2569 factory.go:55] Registering systemd factory
Jul 13 18:49:38.904613 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.904548    2569 factory.go:223] Registration of the systemd container factory successfully
Jul 13 18:49:38.905585 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.905566    2569 factory.go:153] Registering CRI-O factory
Jul 13 18:49:38.905585 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.905586    2569 factory.go:223] Registration of the crio container factory successfully
Jul 13 18:49:38.905721 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.905641    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 13 18:49:38.905721 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.905669    2569 factory.go:103] Registering Raw factory
Jul 13 18:49:38.905721 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.905700    2569 manager.go:1196] Started watching for new ooms in manager
Jul 13 18:49:38.906105 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.906088    2569 manager.go:349] Starting recovery of all containers
Jul 13 18:49:38.908019 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:38.907988    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-134.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 13 18:49:38.908019 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:38.908018    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 13 18:49:38.909106 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.909079    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vx8l9"
Jul 13 18:49:38.911718 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.911673    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 13 18:49:38.919090 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.918922    2569 manager.go:354] Recovery completed
Jul 13 18:49:38.923618 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.923601    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:49:38.925511 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.925492    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-134.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:49:38.925610 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.925527    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-134.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:49:38.925610 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.925542    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-134.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:49:38.926093 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.926078    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 13 18:49:38.926093 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.926091    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 13 18:49:38.926198 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.926109    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 18:49:38.926198 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.926116    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:49:38.926946 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.926933    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-134.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:49:38.927007 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.926958    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-134.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:49:38.927007 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.926982    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-134.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:49:38.928000 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:38.927896    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-134.ec2.internal.18c1edf87d50404f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-134.ec2.internal,UID:ip-10-0-139-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-134.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-134.ec2.internal,},FirstTimestamp:2026-07-13 18:49:38.925510735 +0000 UTC m=+0.439309896,LastTimestamp:2026-07-13 18:49:38.925510735 +0000 UTC m=+0.439309896,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-134.ec2.internal,}"
Jul 13 18:49:38.928347 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.928331    2569 policy_none.go:49] "None policy: Start"
Jul 13 18:49:38.928347 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.928350    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 13 18:49:38.928495 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.928364    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 13 18:49:38.930429 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.930414    2569 policy_none.go:47] "Start"
Jul 13 18:49:38.966351 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.966331    2569 manager.go:337] "Starting Device Plugin manager"
Jul 13 18:49:38.989161 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:38.966369    2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 13 18:49:38.989161 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.966379    2569 server.go:87] "Starting device plugin registration server"
Jul 13 18:49:38.989161 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.966646    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 13 18:49:38.989161 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.966656    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 13 18:49:38.989161 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.966731    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 13 18:49:38.989161 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.966817    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 13 18:49:38.989161 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:38.966828    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 13 18:49:38.989161 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:38.967482    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 13 18:49:38.989161 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:38.967534    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-134.ec2.internal\" not found"
Jul 13 18:49:39.054384 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.054346    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 13 18:49:39.054526 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.054399    2569 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 13 18:49:39.054526 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.054426    2569 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 13 18:49:39.054526 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:39.054467    2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 13 18:49:39.057047 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.057022    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:49:39.067410 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.067338    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:49:39.068466 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.068450    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-134.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:49:39.068520 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.068480    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-134.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:49:39.068520 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.068490    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-134.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:49:39.068520 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.068517    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-134.ec2.internal"
Jul 13 18:49:39.081724 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.081705    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-134.ec2.internal"
Jul 13 18:49:39.081786 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:39.081734    2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-134.ec2.internal\": node \"ip-10-0-139-134.ec2.internal\" not found"
Jul 13 18:49:39.096881 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:39.096854    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-134.ec2.internal\" not found"
Jul 13 18:49:39.154774 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.154721    2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-134.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-134.ec2.internal"]
Jul 13 18:49:39.154864 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.154820    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:49:39.155863 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.155840    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-134.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:49:39.155969 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.155872    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-134.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:49:39.155969 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.155887    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-134.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:49:39.159241 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.159220    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:49:39.159392 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.159353    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-134.ec2.internal"
Jul 13 18:49:39.159392 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.159390    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:49:39.160077 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.160062    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-134.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:49:39.160077 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.160072    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-134.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:49:39.160167 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.160095    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-134.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:49:39.160167 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.160095    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-134.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:49:39.160167 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.160110    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-134.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:49:39.160167 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.160119    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-134.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:49:39.162480 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.162464    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-134.ec2.internal"
Jul 13 18:49:39.162541 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.162491    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:49:39.163199 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.163186    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-134.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:49:39.163282 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.163218    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-134.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:49:39.163282 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.163233    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-134.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:49:39.188008 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:39.187981    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-134.ec2.internal\" not found" node="ip-10-0-139-134.ec2.internal"
Jul 13 18:49:39.192472 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:39.192449    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-134.ec2.internal\" not found" node="ip-10-0-139-134.ec2.internal"
Jul 13 18:49:39.197511 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:39.197492    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-134.ec2.internal\" not found"
Jul 13 18:49:39.204867 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.204846    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6c8631cd46f5ef2fe2e7fec413f6f8d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-134.ec2.internal\" (UID: \"6c8631cd46f5ef2fe2e7fec413f6f8d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-134.ec2.internal"
Jul 13 18:49:39.204972 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.204873    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c8631cd46f5ef2fe2e7fec413f6f8d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-134.ec2.internal\" (UID: \"6c8631cd46f5ef2fe2e7fec413f6f8d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-134.ec2.internal"
Jul 13 18:49:39.204972 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.204892    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d1a4d271968fb37c56f11e30e3e00f8-config\") pod \"kube-apiserver-proxy-ip-10-0-139-134.ec2.internal\" (UID: \"6d1a4d271968fb37c56f11e30e3e00f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-134.ec2.internal"
Jul 13 18:49:39.298311 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:39.298256    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-134.ec2.internal\" not found"
Jul 13 18:49:39.305713 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.305689    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6c8631cd46f5ef2fe2e7fec413f6f8d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-134.ec2.internal\" (UID: \"6c8631cd46f5ef2fe2e7fec413f6f8d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-134.ec2.internal"
Jul 13 18:49:39.305810 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.305723    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c8631cd46f5ef2fe2e7fec413f6f8d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-134.ec2.internal\" (UID: \"6c8631cd46f5ef2fe2e7fec413f6f8d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-134.ec2.internal"
Jul 13 18:49:39.305810 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.305756    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d1a4d271968fb37c56f11e30e3e00f8-config\") pod \"kube-apiserver-proxy-ip-10-0-139-134.ec2.internal\" (UID: \"6d1a4d271968fb37c56f11e30e3e00f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-134.ec2.internal"
Jul 13 18:49:39.305810 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.305805    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d1a4d271968fb37c56f11e30e3e00f8-config\") pod \"kube-apiserver-proxy-ip-10-0-139-134.ec2.internal\" (UID: \"6d1a4d271968fb37c56f11e30e3e00f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-134.ec2.internal"
Jul 13 18:49:39.305933 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.305799    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6c8631cd46f5ef2fe2e7fec413f6f8d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-134.ec2.internal\" (UID: \"6c8631cd46f5ef2fe2e7fec413f6f8d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-134.ec2.internal"
Jul 13 18:49:39.305933 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.305805    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c8631cd46f5ef2fe2e7fec413f6f8d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-134.ec2.internal\" (UID: \"6c8631cd46f5ef2fe2e7fec413f6f8d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-134.ec2.internal"
Jul 13 18:49:39.398722 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:39.398624    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-134.ec2.internal\" not found"
Jul 13 18:49:39.490240 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.490185    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-134.ec2.internal"
Jul 13 18:49:39.495484 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.495134    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-134.ec2.internal"
Jul 13 18:49:39.498761 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:39.498736    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-134.ec2.internal\" not found"
Jul 13 18:49:39.599616 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:39.599573    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-134.ec2.internal\" not found"
Jul 13 18:49:39.700182 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:39.700090    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-134.ec2.internal\" not found"
Jul 13 18:49:39.796215 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.796188    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:49:39.800314 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:39.800283    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-134.ec2.internal\" not found"
Jul 13 18:49:39.821915 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.821886    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:49:39.835001 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.834984    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 13 18:49:39.835144 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.835124    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 18:49:39.835186 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.835155    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 18:49:39.835186 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.835171    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 18:49:39.900578 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:39.900526    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-134.ec2.internal\" not found"
Jul 13 18:49:39.915455 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.915418    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 18:44:39 +0000 UTC" deadline="2028-02-16 12:16:52.280335745 +0000 UTC"
Jul 13 18:49:39.915455 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:39.915449    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13985h27m12.364889913s"
Jul 13 18:49:40.000846 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:40.000761    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-134.ec2.internal\" not found"
Jul 13 18:49:40.101026 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:40.100994    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-134.ec2.internal\" not found"
Jul 13 18:49:40.185864 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:40.185831    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8631cd46f5ef2fe2e7fec413f6f8d7.slice/crio-b0eb66b44d1b86997a7ffccfb2f4873df7f7b3587cf6a1b61063a59c221cdda6 WatchSource:0}: Error finding container b0eb66b44d1b86997a7ffccfb2f4873df7f7b3587cf6a1b61063a59c221cdda6: Status 404 returned error can't find the container with id b0eb66b44d1b86997a7ffccfb2f4873df7f7b3587cf6a1b61063a59c221cdda6
Jul 13 18:49:40.190936 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.190917    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 18:49:40.201503 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:40.201483    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-134.ec2.internal\" not found"
Jul 13 18:49:40.309257 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:40.309162    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-134.ec2.internal\" not found"
Jul 13 18:49:40.364646 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.364603    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:49:40.371471 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:40.371444    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d1a4d271968fb37c56f11e30e3e00f8.slice/crio-bb28f57fb9747aef18d8836568d83edb632ae62a961dea4898e0719eb1aa0885 WatchSource:0}: Error finding container bb28f57fb9747aef18d8836568d83edb632ae62a961dea4898e0719eb1aa0885: Status 404 returned error can't find the container with id bb28f57fb9747aef18d8836568d83edb632ae62a961dea4898e0719eb1aa0885
Jul 13 18:49:40.402374 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.402335    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-134.ec2.internal"
Jul 13 18:49:40.416065 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.416035    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 18:49:40.416915 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.416897    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-134.ec2.internal"
Jul 13 18:49:40.425021 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.424989    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 18:49:40.672523 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.672434    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:49:40.882796 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.882765    2569 apiserver.go:52] "Watching apiserver"
Jul 13 18:49:40.895186 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.895153    2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 13 18:49:40.895627 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.895600    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-bmcpv","openshift-image-registry/node-ca-gdf7f","openshift-multus/multus-bmsb4","openshift-network-diagnostics/network-check-target-xrtxq","openshift-ovn-kubernetes/ovnkube-node-v76kh","kube-system/konnectivity-agent-z9pwv","kube-system/kube-apiserver-proxy-ip-10-0-139-134.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h","openshift-dns/node-resolver-vxf4w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-134.ec2.internal","openshift-multus/multus-additional-cni-plugins-bc97r","openshift-multus/network-metrics-daemon-xnfv2","openshift-network-operator/iptables-alerter-j7qjq","kube-system/global-pull-secret-syncer-66pbz"]
Jul 13 18:49:40.898348 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.898318    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j7qjq"
Jul 13 18:49:40.900937 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.900903    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gdf7f"
Jul 13 18:49:40.902837 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.902812    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 13 18:49:40.904270 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.904244    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:40.904891 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.904865    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 13 18:49:40.905020 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.904895    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p699b\""
Jul 13 18:49:40.905020 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.904948    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 13 18:49:40.905020 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.904954    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 13 18:49:40.905020 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.904997    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 13 18:49:40.905268 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.905242    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 13 18:49:40.906439 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.906419    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 13 18:49:40.907571 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.907550    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9gbfj\""
Jul 13 18:49:40.908012 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.907893    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 13 18:49:40.908155 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.908139    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 13 18:49:40.908225 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.908193    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:49:40.908758 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.908734    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 13 18:49:40.909615 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.909596    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7v5xt\""
Jul 13 18:49:40.909870 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.909854    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 13 18:49:40.911092 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:40.911055    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrtxq" podUID="6d8ac836-95a6-49cf-85ba-99e3db827c52"
Jul 13 18:49:40.911239 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.911100    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:40.913573 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.913552    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z9pwv"
Jul 13 18:49:40.915477 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.915457    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ddda9308-3395-4d7b-af21-a398125bd512-host\") pod \"node-ca-gdf7f\" (UID: \"ddda9308-3395-4d7b-af21-a398125bd512\") " pod="openshift-image-registry/node-ca-gdf7f"
Jul 13 18:49:40.915583 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.915485    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-host-var-lib-kubelet\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:40.915583 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.915503    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-hostroot\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:40.915583 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.915521    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc2tc\" (UniqueName: \"kubernetes.io/projected/ddda9308-3395-4d7b-af21-a398125bd512-kube-api-access-xc2tc\") pod \"node-ca-gdf7f\" (UID: \"ddda9308-3395-4d7b-af21-a398125bd512\") " pod="openshift-image-registry/node-ca-gdf7f"
Jul 13 18:49:40.915583 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.915546    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-multus-socket-dir-parent\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:40.915583 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.915568    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-host-var-lib-cni-bin\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:40.915798 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.915621    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-host-run-multus-certs\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:40.915798 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.915677    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-etc-kubernetes\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:40.915798 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.915721    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppck7\" (UniqueName: \"kubernetes.io/projected/898f3564-8c07-40c7-9580-70071f193268-kube-api-access-ppck7\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:40.915798 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.915773    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxc6f\" (UniqueName: \"kubernetes.io/projected/e1baa76b-d5eb-43dc-9c73-6b4fa9f2989b-kube-api-access-dxc6f\") pod \"iptables-alerter-j7qjq\" (UID: \"e1baa76b-d5eb-43dc-9c73-6b4fa9f2989b\") " pod="openshift-network-operator/iptables-alerter-j7qjq"
Jul 13 18:49:40.915961 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.915799    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ddda9308-3395-4d7b-af21-a398125bd512-serviceca\") pod \"node-ca-gdf7f\" (UID: \"ddda9308-3395-4d7b-af21-a398125bd512\") " pod="openshift-image-registry/node-ca-gdf7f"
Jul 13 18:49:40.915961 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.915836    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-system-cni-dir\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:40.915961 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.915870    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-multus-cni-dir\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:40.915961 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.915923    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/898f3564-8c07-40c7-9580-70071f193268-cni-binary-copy\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:40.915961 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.915954    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-host-run-netns\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:40.916200 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.915983    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1baa76b-d5eb-43dc-9c73-6b4fa9f2989b-iptables-alerter-script\") pod \"iptables-alerter-j7qjq\" (UID: \"e1baa76b-d5eb-43dc-9c73-6b4fa9f2989b\") " pod="openshift-network-operator/iptables-alerter-j7qjq"
Jul 13 18:49:40.916200 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.916028    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1baa76b-d5eb-43dc-9c73-6b4fa9f2989b-host-slash\") pod \"iptables-alerter-j7qjq\" (UID: \"e1baa76b-d5eb-43dc-9c73-6b4fa9f2989b\") " pod="openshift-network-operator/iptables-alerter-j7qjq"
Jul 13 18:49:40.916200 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.916052    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-cnibin\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:40.916200 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.916074    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-os-release\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:40.916200 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.916104    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-host-run-k8s-cni-cncf-io\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:40.916200 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.916118    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 13 18:49:40.916200 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.916128    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-host-var-lib-cni-multus\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:40.916200 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.916152    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-multus-conf-dir\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:40.916200 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.916197    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/898f3564-8c07-40c7-9580-70071f193268-multus-daemon-config\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:40.916674 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.916376    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:40.917536 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.917516    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 13 18:49:40.918774 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.918753    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 13 18:49:40.918774 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.918761    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 13 18:49:40.918918 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.918894    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 13 18:49:40.919008 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.918992    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 13 18:49:40.919081 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.919064    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 13 18:49:40.919141 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.919122    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p96qv\""
Jul 13 18:49:40.919260 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.919244    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 13 18:49:40.919327 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.919299    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:40.920349 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.920327    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9prxv\""
Jul 13 18:49:40.920450 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.920347    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 13 18:49:40.920450 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.920349    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rwd9x\""
Jul 13 18:49:40.920645 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.920630    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 13 18:49:40.921904 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.921884    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 13 18:49:40.922577 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.922524    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vxf4w"
Jul 13 18:49:40.922951 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.922928    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 18:49:40.923083 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.923034    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8ght8\""
Jul 13 18:49:40.923174 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.923109    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 13 18:49:40.923411 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.923396    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 13 18:49:40.925248 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.925230    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:40.925337 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:40.925289    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66pbz" podUID="b1f63a2f-aa86-415d-8c72-78c96c560625"
Jul 13 18:49:40.925641 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.925624    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 13 18:49:40.926175 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.925954    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 13 18:49:40.926175 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.925982    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-47xtd\""
Jul 13 18:49:40.927952 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.927930    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:40.930285 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.930265    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:49:40.930390 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:40.930368    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnfv2" podUID="6ec3b9c9-8324-4d12-92b8-6baa78773fe2"
Jul 13 18:49:40.930788 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.930768    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 13 18:49:40.930788 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.930782    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qgxdh\""
Jul 13 18:49:40.930938 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.930808    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 13 18:49:40.965532 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.965505    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-crswr"
Jul 13 18:49:40.989362 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:40.989327    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-crswr"
Jul 13 18:49:41.003836 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.003807    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 13 18:49:41.022758 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.019549    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-etc-kubernetes\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.022758 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.019614    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-etc-sysctl-conf\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.022758 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.019648    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-tmp\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.022758 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.019680    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eacba2f0-6acf-4ce6-a960-0f5b0c8a77d9-konnectivity-ca\") pod \"konnectivity-agent-z9pwv\" (UID: \"eacba2f0-6acf-4ce6-a960-0f5b0c8a77d9\") " pod="kube-system/konnectivity-agent-z9pwv"
Jul 13 18:49:41.022758 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.019711    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/658136e2-705b-4244-97e1-ca2913f0474f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b247h\" (UID: \"658136e2-705b-4244-97e1-ca2913f0474f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:41.022758 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.019748    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppck7\" (UniqueName: \"kubernetes.io/projected/898f3564-8c07-40c7-9580-70071f193268-kube-api-access-ppck7\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.022758 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.019782    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsw7c\" (UniqueName: \"kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c\") pod \"network-check-target-xrtxq\" (UID: \"6d8ac836-95a6-49cf-85ba-99e3db827c52\") " pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:49:41.022758 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.019814    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-host-run-netns\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.022758 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.019843    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d254ad6f-c371-4b41-a97d-97056891fdf6-ovnkube-config\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.022758 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.020324    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/658136e2-705b-4244-97e1-ca2913f0474f-etc-selinux\") pod \"aws-ebs-csi-driver-node-b247h\" (UID: \"658136e2-705b-4244-97e1-ca2913f0474f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:41.022758 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.020357    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3963e434-e4ab-4d73-b496-2472c7711891-tmp-dir\") pod \"node-resolver-vxf4w\" (UID: \"3963e434-e4ab-4d73-b496-2472c7711891\") " pod="openshift-dns/node-resolver-vxf4w"
Jul 13 18:49:41.022758 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.020417    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxc6f\" (UniqueName: \"kubernetes.io/projected/e1baa76b-d5eb-43dc-9c73-6b4fa9f2989b-kube-api-access-dxc6f\") pod \"iptables-alerter-j7qjq\" (UID: \"e1baa76b-d5eb-43dc-9c73-6b4fa9f2989b\") " pod="openshift-network-operator/iptables-alerter-j7qjq"
Jul 13 18:49:41.022758 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.020450    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-host-kubelet\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.022758 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.020473    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-node-log\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.022758 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.020502    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d254ad6f-c371-4b41-a97d-97056891fdf6-env-overrides\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.022758 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.020530    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-etc-tuned\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.023628 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.020562    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/658136e2-705b-4244-97e1-ca2913f0474f-device-dir\") pod \"aws-ebs-csi-driver-node-b247h\" (UID: \"658136e2-705b-4244-97e1-ca2913f0474f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:41.023628 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.020601    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1f63a2f-aa86-415d-8c72-78c96c560625-kubelet-config\") pod \"global-pull-secret-syncer-66pbz\" (UID: \"b1f63a2f-aa86-415d-8c72-78c96c560625\") " pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:41.023628 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.020629    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-cnibin\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.023628 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.020662    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-var-lib-openvswitch\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.023628 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.020695    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdz5z\" (UniqueName: \"kubernetes.io/projected/d254ad6f-c371-4b41-a97d-97056891fdf6-kube-api-access-cdz5z\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.023628 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.020724    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-sys\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.023628 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.020756    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eacba2f0-6acf-4ce6-a960-0f5b0c8a77d9-agent-certs\") pod \"konnectivity-agent-z9pwv\" (UID: \"eacba2f0-6acf-4ce6-a960-0f5b0c8a77d9\") " pod="kube-system/konnectivity-agent-z9pwv"
Jul 13 18:49:41.023628 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.020780    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-etc-sysconfig\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.023628 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.020810    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-etc-systemd\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.023628 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.020838    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/658136e2-705b-4244-97e1-ca2913f0474f-sys-fs\") pod \"aws-ebs-csi-driver-node-b247h\" (UID: \"658136e2-705b-4244-97e1-ca2913f0474f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:41.023628 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.020867    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret\") pod \"global-pull-secret-syncer-66pbz\" (UID: \"b1f63a2f-aa86-415d-8c72-78c96c560625\") " pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:41.023628 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.020897    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-run\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.023628 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.020920    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/658136e2-705b-4244-97e1-ca2913f0474f-registration-dir\") pod \"aws-ebs-csi-driver-node-b247h\" (UID: \"658136e2-705b-4244-97e1-ca2913f0474f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:41.023628 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.020965    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.023628 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.021013    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/898f3564-8c07-40c7-9580-70071f193268-cni-binary-copy\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.023628 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.021129    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-etc-openvswitch\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.024321 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.021148    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d254ad6f-c371-4b41-a97d-97056891fdf6-ovnkube-script-lib\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.024321 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.021174    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1baa76b-d5eb-43dc-9c73-6b4fa9f2989b-host-slash\") pod \"iptables-alerter-j7qjq\" (UID: \"e1baa76b-d5eb-43dc-9c73-6b4fa9f2989b\") " pod="openshift-network-operator/iptables-alerter-j7qjq"
Jul 13 18:49:41.024321 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.021197    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-host-run-ovn-kubernetes\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.024321 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.021277    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-cnibin\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.024321 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.021334    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1baa76b-d5eb-43dc-9c73-6b4fa9f2989b-host-slash\") pod \"iptables-alerter-j7qjq\" (UID: \"e1baa76b-d5eb-43dc-9c73-6b4fa9f2989b\") " pod="openshift-network-operator/iptables-alerter-j7qjq"
Jul 13 18:49:41.024321 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.021387    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-system-cni-dir\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.024321 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.021437    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.024321 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.021480    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7swk\" (UniqueName: \"kubernetes.io/projected/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-kube-api-access-t7swk\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.024321 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.021518    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-host-var-lib-kubelet\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.024321 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.021553    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxpdj\" (UniqueName: \"kubernetes.io/projected/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-kube-api-access-xxpdj\") pod \"network-metrics-daemon-xnfv2\" (UID: \"6ec3b9c9-8324-4d12-92b8-6baa78773fe2\") " pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:49:41.024321 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.021581    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-host\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.024321 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.021618    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-host-var-lib-cni-bin\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.024321 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.021747    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d254ad6f-c371-4b41-a97d-97056891fdf6-ovn-node-metrics-cert\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.024321 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.021804    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1f63a2f-aa86-415d-8c72-78c96c560625-dbus\") pod \"global-pull-secret-syncer-66pbz\" (UID: \"b1f63a2f-aa86-415d-8c72-78c96c560625\") " pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:41.024321 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.021810    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-host-var-lib-cni-bin\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.024321 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.021832    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/658136e2-705b-4244-97e1-ca2913f0474f-socket-dir\") pod \"aws-ebs-csi-driver-node-b247h\" (UID: \"658136e2-705b-4244-97e1-ca2913f0474f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:41.024321 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.021851    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/898f3564-8c07-40c7-9580-70071f193268-cni-binary-copy\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.024999 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022116    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ddda9308-3395-4d7b-af21-a398125bd512-serviceca\") pod \"node-ca-gdf7f\" (UID: \"ddda9308-3395-4d7b-af21-a398125bd512\") " pod="openshift-image-registry/node-ca-gdf7f"
Jul 13 18:49:41.024999 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022138    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-system-cni-dir\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.024999 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022209    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-multus-cni-dir\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.024999 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022237    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-host-var-lib-cni-multus\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.024999 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022303    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-log-socket\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.024999 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022306    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-host-var-lib-kubelet\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.024999 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022321    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-host-var-lib-cni-multus\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.024999 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022381    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-system-cni-dir\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.024999 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022416    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.024999 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022451    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-multus-cni-dir\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.024999 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022461    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-lib-modules\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.024999 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022544    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3963e434-e4ab-4d73-b496-2472c7711891-hosts-file\") pod \"node-resolver-vxf4w\" (UID: \"3963e434-e4ab-4d73-b496-2472c7711891\") " pod="openshift-dns/node-resolver-vxf4w"
Jul 13 18:49:41.024999 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022573    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs\") pod \"network-metrics-daemon-xnfv2\" (UID: \"6ec3b9c9-8324-4d12-92b8-6baa78773fe2\") " pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:49:41.024999 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022607    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-os-release\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.024999 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022639    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-multus-conf-dir\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.024999 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022671    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-run-systemd\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.024999 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022703    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.025868 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022719    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ddda9308-3395-4d7b-af21-a398125bd512-serviceca\") pod \"node-ca-gdf7f\" (UID: \"ddda9308-3395-4d7b-af21-a398125bd512\") " pod="openshift-image-registry/node-ca-gdf7f"
Jul 13 18:49:41.025868 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022757    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-os-release\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.025868 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022757    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-multus-conf-dir\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.025868 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022802    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g567s\" (UniqueName: \"kubernetes.io/projected/658136e2-705b-4244-97e1-ca2913f0474f-kube-api-access-g567s\") pod \"aws-ebs-csi-driver-node-b247h\" (UID: \"658136e2-705b-4244-97e1-ca2913f0474f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:41.025868 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022887    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q265b\" (UniqueName: \"kubernetes.io/projected/3963e434-e4ab-4d73-b496-2472c7711891-kube-api-access-q265b\") pod \"node-resolver-vxf4w\" (UID: \"3963e434-e4ab-4d73-b496-2472c7711891\") " pod="openshift-dns/node-resolver-vxf4w"
Jul 13 18:49:41.025868 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022924    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ddda9308-3395-4d7b-af21-a398125bd512-host\") pod \"node-ca-gdf7f\" (UID: \"ddda9308-3395-4d7b-af21-a398125bd512\") " pod="openshift-image-registry/node-ca-gdf7f"
Jul 13 18:49:41.025868 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.022994    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-hostroot\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.025868 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023017    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-hostroot\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.025868 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023055    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-run-openvswitch\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.025868 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023077    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-os-release\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.025868 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023101    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc2tc\" (UniqueName: \"kubernetes.io/projected/ddda9308-3395-4d7b-af21-a398125bd512-kube-api-access-xc2tc\") pod \"node-ca-gdf7f\" (UID: \"ddda9308-3395-4d7b-af21-a398125bd512\") " pod="openshift-image-registry/node-ca-gdf7f"
Jul 13 18:49:41.025868 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023054    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ddda9308-3395-4d7b-af21-a398125bd512-host\") pod \"node-ca-gdf7f\" (UID: \"ddda9308-3395-4d7b-af21-a398125bd512\") " pod="openshift-image-registry/node-ca-gdf7f"
Jul 13 18:49:41.025868 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023125    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-multus-socket-dir-parent\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.025868 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023189    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-host-run-multus-certs\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.025868 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023224    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-systemd-units\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.025868 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023228    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-multus-socket-dir-parent\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.025868 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023242    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-cnibin\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.026601 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023265    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnn64\" (UniqueName: \"kubernetes.io/projected/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-kube-api-access-cnn64\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.026601 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023345    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-etc-modprobe-d\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.026601 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023372    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-host-run-multus-certs\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.026601 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023380    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-var-lib-kubelet\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.026601 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023425    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-etc-kubernetes\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.026601 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023489    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-etc-kubernetes\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.026601 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023511    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-host-run-netns\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.026601 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023545    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/898f3564-8c07-40c7-9580-70071f193268-multus-daemon-config\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.026601 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023589    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-host-run-netns\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.026601 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023636    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-run-ovn\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.026601 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023681    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-host-cni-bin\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.026601 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023707    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-etc-sysctl-d\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.026601 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023743    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1baa76b-d5eb-43dc-9c73-6b4fa9f2989b-iptables-alerter-script\") pod \"iptables-alerter-j7qjq\" (UID: \"e1baa76b-d5eb-43dc-9c73-6b4fa9f2989b\") " pod="openshift-network-operator/iptables-alerter-j7qjq"
Jul 13 18:49:41.026601 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023778    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-host-run-k8s-cni-cncf-io\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.026601 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023816    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-host-slash\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.026601 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023838    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/898f3564-8c07-40c7-9580-70071f193268-host-run-k8s-cni-cncf-io\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.026601 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023843    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-host-cni-netd\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.027314 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.023901    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.027314 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.024088    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/898f3564-8c07-40c7-9580-70071f193268-multus-daemon-config\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.027314 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.024347    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1baa76b-d5eb-43dc-9c73-6b4fa9f2989b-iptables-alerter-script\") pod \"iptables-alerter-j7qjq\" (UID: \"e1baa76b-d5eb-43dc-9c73-6b4fa9f2989b\") " pod="openshift-network-operator/iptables-alerter-j7qjq"
Jul 13 18:49:41.029174 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.029146    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 13 18:49:41.032959 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.032935    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppck7\" (UniqueName: \"kubernetes.io/projected/898f3564-8c07-40c7-9580-70071f193268-kube-api-access-ppck7\") pod \"multus-bmsb4\" (UID: \"898f3564-8c07-40c7-9580-70071f193268\") " pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.033103 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.032968    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxc6f\" (UniqueName: \"kubernetes.io/projected/e1baa76b-d5eb-43dc-9c73-6b4fa9f2989b-kube-api-access-dxc6f\") pod \"iptables-alerter-j7qjq\" (UID: \"e1baa76b-d5eb-43dc-9c73-6b4fa9f2989b\") " pod="openshift-network-operator/iptables-alerter-j7qjq"
Jul 13 18:49:41.033242 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.033225    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc2tc\" (UniqueName: \"kubernetes.io/projected/ddda9308-3395-4d7b-af21-a398125bd512-kube-api-access-xc2tc\") pod \"node-ca-gdf7f\" (UID: \"ddda9308-3395-4d7b-af21-a398125bd512\") " pod="openshift-image-registry/node-ca-gdf7f"
Jul 13 18:49:41.060127 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.060069    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-134.ec2.internal" event={"ID":"6d1a4d271968fb37c56f11e30e3e00f8","Type":"ContainerStarted","Data":"bb28f57fb9747aef18d8836568d83edb632ae62a961dea4898e0719eb1aa0885"}
Jul 13 18:49:41.061418 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.061391    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-134.ec2.internal" event={"ID":"6c8631cd46f5ef2fe2e7fec413f6f8d7","Type":"ContainerStarted","Data":"b0eb66b44d1b86997a7ffccfb2f4873df7f7b3587cf6a1b61063a59c221cdda6"}
Jul 13 18:49:41.124738 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.124677    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.124738 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.124723    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-etc-openvswitch\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.124964 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.124750    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d254ad6f-c371-4b41-a97d-97056891fdf6-ovnkube-script-lib\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.124964 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.124774    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-host-run-ovn-kubernetes\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.124964 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.124799    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-system-cni-dir\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.124964 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.124825    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.124964 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.124849    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7swk\" (UniqueName: \"kubernetes.io/projected/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-kube-api-access-t7swk\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.124964 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.124875    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxpdj\" (UniqueName: \"kubernetes.io/projected/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-kube-api-access-xxpdj\") pod \"network-metrics-daemon-xnfv2\" (UID: \"6ec3b9c9-8324-4d12-92b8-6baa78773fe2\") " pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:49:41.124964 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.124899    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-host\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.124964 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.124928    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d254ad6f-c371-4b41-a97d-97056891fdf6-ovn-node-metrics-cert\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.124964 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.124957    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1f63a2f-aa86-415d-8c72-78c96c560625-dbus\") pod \"global-pull-secret-syncer-66pbz\" (UID: \"b1f63a2f-aa86-415d-8c72-78c96c560625\") " pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:41.125258 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.124989    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/658136e2-705b-4244-97e1-ca2913f0474f-socket-dir\") pod \"aws-ebs-csi-driver-node-b247h\" (UID: \"658136e2-705b-4244-97e1-ca2913f0474f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:41.125258 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125020    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-log-socket\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.125258 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125046    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.125258 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125069    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-lib-modules\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.125258 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125093    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3963e434-e4ab-4d73-b496-2472c7711891-hosts-file\") pod \"node-resolver-vxf4w\" (UID: \"3963e434-e4ab-4d73-b496-2472c7711891\") " pod="openshift-dns/node-resolver-vxf4w"
Jul 13 18:49:41.125258 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125118    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs\") pod \"network-metrics-daemon-xnfv2\" (UID: \"6ec3b9c9-8324-4d12-92b8-6baa78773fe2\") " pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:49:41.125258 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125163    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-run-systemd\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.125258 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125209    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.125258 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125236    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g567s\" (UniqueName: \"kubernetes.io/projected/658136e2-705b-4244-97e1-ca2913f0474f-kube-api-access-g567s\") pod \"aws-ebs-csi-driver-node-b247h\" (UID: \"658136e2-705b-4244-97e1-ca2913f0474f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:41.125568 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125261    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q265b\" (UniqueName: \"kubernetes.io/projected/3963e434-e4ab-4d73-b496-2472c7711891-kube-api-access-q265b\") pod \"node-resolver-vxf4w\" (UID: \"3963e434-e4ab-4d73-b496-2472c7711891\") " pod="openshift-dns/node-resolver-vxf4w"
Jul 13 18:49:41.125568 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125291    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-run-openvswitch\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.125568 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125323    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-os-release\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.125568 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125351    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-systemd-units\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.125568 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125400    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-cnibin\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.125568 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125425    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnn64\" (UniqueName: \"kubernetes.io/projected/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-kube-api-access-cnn64\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.125568 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125461    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-etc-modprobe-d\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.125568 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125498    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-var-lib-kubelet\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.125568 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125528    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-run-ovn\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.125568 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125551    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-host-cni-bin\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.125928 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125575    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-etc-sysctl-d\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.125928 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125614    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-host-slash\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.125928 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125640    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-host-cni-netd\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.125928 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125661    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.125928 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125684    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-etc-kubernetes\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.125928 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125709    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-etc-sysctl-conf\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.125928 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125721    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3963e434-e4ab-4d73-b496-2472c7711891-hosts-file\") pod \"node-resolver-vxf4w\" (UID: \"3963e434-e4ab-4d73-b496-2472c7711891\") " pod="openshift-dns/node-resolver-vxf4w"
Jul 13 18:49:41.125928 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125732    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-tmp\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.125928 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125770    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eacba2f0-6acf-4ce6-a960-0f5b0c8a77d9-konnectivity-ca\") pod \"konnectivity-agent-z9pwv\" (UID: \"eacba2f0-6acf-4ce6-a960-0f5b0c8a77d9\") " pod="kube-system/konnectivity-agent-z9pwv"
Jul 13 18:49:41.125928 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125796    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/658136e2-705b-4244-97e1-ca2913f0474f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b247h\" (UID: \"658136e2-705b-4244-97e1-ca2913f0474f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:41.125928 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125826    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsw7c\" (UniqueName: \"kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c\") pod \"network-check-target-xrtxq\" (UID: \"6d8ac836-95a6-49cf-85ba-99e3db827c52\") " pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:49:41.125928 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125850    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-host-run-netns\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.125928 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125906    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d254ad6f-c371-4b41-a97d-97056891fdf6-ovnkube-config\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.126409 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125947    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/658136e2-705b-4244-97e1-ca2913f0474f-etc-selinux\") pod \"aws-ebs-csi-driver-node-b247h\" (UID: \"658136e2-705b-4244-97e1-ca2913f0474f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:41.126409 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125974    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3963e434-e4ab-4d73-b496-2472c7711891-tmp-dir\") pod \"node-resolver-vxf4w\" (UID: \"3963e434-e4ab-4d73-b496-2472c7711891\") " pod="openshift-dns/node-resolver-vxf4w"
Jul 13 18:49:41.126409 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126002    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-host-kubelet\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.126409 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126028    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-node-log\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.126409 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126057    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d254ad6f-c371-4b41-a97d-97056891fdf6-env-overrides\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.126409 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126081    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-etc-tuned\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.126409 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126106    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/658136e2-705b-4244-97e1-ca2913f0474f-device-dir\") pod \"aws-ebs-csi-driver-node-b247h\" (UID: \"658136e2-705b-4244-97e1-ca2913f0474f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:41.126409 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126135    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1f63a2f-aa86-415d-8c72-78c96c560625-kubelet-config\") pod \"global-pull-secret-syncer-66pbz\" (UID: \"b1f63a2f-aa86-415d-8c72-78c96c560625\") " pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:41.126409 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126190    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-system-cni-dir\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.126409 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126195    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1f63a2f-aa86-415d-8c72-78c96c560625-kubelet-config\") pod \"global-pull-secret-syncer-66pbz\" (UID: \"b1f63a2f-aa86-415d-8c72-78c96c560625\") " pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:41.126409 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126218    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-host-run-ovn-kubernetes\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.126409 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126242    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-host-run-netns\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.126409 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126254    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-host-kubelet\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.126409 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126270    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-var-lib-openvswitch\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.126409 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126320    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.126409 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.125324    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.126409 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126418    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-lib-modules\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.127299 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126461    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-var-lib-openvswitch\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.127299 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126544    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1f63a2f-aa86-415d-8c72-78c96c560625-dbus\") pod \"global-pull-secret-syncer-66pbz\" (UID: \"b1f63a2f-aa86-415d-8c72-78c96c560625\") " pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:41.127299 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126615    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-run-openvswitch\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.127299 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126650    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-node-log\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.127299 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126753    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-cnibin\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.127299 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126845    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-etc-sysctl-d\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.127299 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126847    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-host\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.127299 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126875    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-etc-openvswitch\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.127299 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126926    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-etc-modprobe-d\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.127299 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:41.126935    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:41.127299 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126956    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-run-ovn\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.127299 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.126988    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-host-cni-bin\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.127299 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:41.127030    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs podName:6ec3b9c9-8324-4d12-92b8-6baa78773fe2 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:41.626981391 +0000 UTC m=+3.140780537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs") pod "network-metrics-daemon-xnfv2" (UID: "6ec3b9c9-8324-4d12-92b8-6baa78773fe2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:41.127299 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127033    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-var-lib-kubelet\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.127299 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127069    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.127299 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127109    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-etc-kubernetes\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.127299 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127145    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-host-slash\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.128169 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127152    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d254ad6f-c371-4b41-a97d-97056891fdf6-ovnkube-script-lib\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.128169 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127225    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d254ad6f-c371-4b41-a97d-97056891fdf6-ovnkube-config\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.128169 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127280    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-etc-sysctl-conf\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.128169 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127305    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/658136e2-705b-4244-97e1-ca2913f0474f-device-dir\") pod \"aws-ebs-csi-driver-node-b247h\" (UID: \"658136e2-705b-4244-97e1-ca2913f0474f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:41.128169 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127320    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-host-cni-netd\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.128169 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127354    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.128169 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127371    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdz5z\" (UniqueName: \"kubernetes.io/projected/d254ad6f-c371-4b41-a97d-97056891fdf6-kube-api-access-cdz5z\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.128169 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127399    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-sys\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.128169 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127419    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/658136e2-705b-4244-97e1-ca2913f0474f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b247h\" (UID: \"658136e2-705b-4244-97e1-ca2913f0474f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:41.128169 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127429    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eacba2f0-6acf-4ce6-a960-0f5b0c8a77d9-agent-certs\") pod \"konnectivity-agent-z9pwv\" (UID: \"eacba2f0-6acf-4ce6-a960-0f5b0c8a77d9\") " pod="kube-system/konnectivity-agent-z9pwv"
Jul 13 18:49:41.128169 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127433    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-systemd-units\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.128169 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127449    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-etc-sysconfig\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.128169 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127457    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-run-systemd\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.128169 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127472    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-etc-systemd\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.128169 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127473    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d254ad6f-c371-4b41-a97d-97056891fdf6-log-socket\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.128169 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127400    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-os-release\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.128169 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127504    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/658136e2-705b-4244-97e1-ca2913f0474f-etc-selinux\") pod \"aws-ebs-csi-driver-node-b247h\" (UID: \"658136e2-705b-4244-97e1-ca2913f0474f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:41.129233 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127706    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/658136e2-705b-4244-97e1-ca2913f0474f-sys-fs\") pod \"aws-ebs-csi-driver-node-b247h\" (UID: \"658136e2-705b-4244-97e1-ca2913f0474f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:41.129233 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127740    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret\") pod \"global-pull-secret-syncer-66pbz\" (UID: \"b1f63a2f-aa86-415d-8c72-78c96c560625\") " pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:41.129233 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127755    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3963e434-e4ab-4d73-b496-2472c7711891-tmp-dir\") pod \"node-resolver-vxf4w\" (UID: \"3963e434-e4ab-4d73-b496-2472c7711891\") " pod="openshift-dns/node-resolver-vxf4w"
Jul 13 18:49:41.129233 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127768    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-run\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.129233 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127799    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/658136e2-705b-4244-97e1-ca2913f0474f-registration-dir\") pod \"aws-ebs-csi-driver-node-b247h\" (UID: \"658136e2-705b-4244-97e1-ca2913f0474f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:41.129233 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127895    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/658136e2-705b-4244-97e1-ca2913f0474f-registration-dir\") pod \"aws-ebs-csi-driver-node-b247h\" (UID: \"658136e2-705b-4244-97e1-ca2913f0474f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:41.129233 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127936    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d254ad6f-c371-4b41-a97d-97056891fdf6-env-overrides\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.129233 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127954    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.129233 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.127987    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/658136e2-705b-4244-97e1-ca2913f0474f-socket-dir\") pod \"aws-ebs-csi-driver-node-b247h\" (UID: \"658136e2-705b-4244-97e1-ca2913f0474f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:41.129233 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.128049    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-sys\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.129233 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.128142    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eacba2f0-6acf-4ce6-a960-0f5b0c8a77d9-konnectivity-ca\") pod \"konnectivity-agent-z9pwv\" (UID: \"eacba2f0-6acf-4ce6-a960-0f5b0c8a77d9\") " pod="kube-system/konnectivity-agent-z9pwv"
Jul 13 18:49:41.129233 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.128256    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-etc-systemd\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.129233 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:41.128291    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:41.129233 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:41.128338    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret podName:b1f63a2f-aa86-415d-8c72-78c96c560625 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:41.628324064 +0000 UTC m=+3.142123209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret") pod "global-pull-secret-syncer-66pbz" (UID: "b1f63a2f-aa86-415d-8c72-78c96c560625") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:41.129233 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.128403    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/658136e2-705b-4244-97e1-ca2913f0474f-sys-fs\") pod \"aws-ebs-csi-driver-node-b247h\" (UID: \"658136e2-705b-4244-97e1-ca2913f0474f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:41.129233 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.128430    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-etc-sysconfig\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.129233 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.128483    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-run\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.130189 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.128834    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-tmp\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.132898 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.132868    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d254ad6f-c371-4b41-a97d-97056891fdf6-ovn-node-metrics-cert\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.133025 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.132966    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eacba2f0-6acf-4ce6-a960-0f5b0c8a77d9-agent-certs\") pod \"konnectivity-agent-z9pwv\" (UID: \"eacba2f0-6acf-4ce6-a960-0f5b0c8a77d9\") " pod="kube-system/konnectivity-agent-z9pwv"
Jul 13 18:49:41.133177 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.133148    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-etc-tuned\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.136100 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:41.135566    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:49:41.136100 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:41.135594    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:49:41.136100 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:41.135608    2569 projected.go:196] Error preparing data for projected volume kube-api-access-vsw7c for pod openshift-network-diagnostics/network-check-target-xrtxq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:41.136100 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:41.135674    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c podName:6d8ac836-95a6-49cf-85ba-99e3db827c52 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:41.635655979 +0000 UTC m=+3.149455131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vsw7c" (UniqueName: "kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c") pod "network-check-target-xrtxq" (UID: "6d8ac836-95a6-49cf-85ba-99e3db827c52") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:41.138388 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.137979    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxpdj\" (UniqueName: \"kubernetes.io/projected/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-kube-api-access-xxpdj\") pod \"network-metrics-daemon-xnfv2\" (UID: \"6ec3b9c9-8324-4d12-92b8-6baa78773fe2\") " pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:49:41.140874 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.140750    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7swk\" (UniqueName: \"kubernetes.io/projected/fcfca38c-2dcb-4e9d-9c13-38927d35a19f-kube-api-access-t7swk\") pod \"tuned-bmcpv\" (UID: \"fcfca38c-2dcb-4e9d-9c13-38927d35a19f\") " pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.140977 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.140964    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnn64\" (UniqueName: \"kubernetes.io/projected/f6c396f9-fccb-492d-9e7e-36c4e864a2d1-kube-api-access-cnn64\") pod \"multus-additional-cni-plugins-bc97r\" (UID: \"f6c396f9-fccb-492d-9e7e-36c4e864a2d1\") " pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.141410 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.141362    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdz5z\" (UniqueName: \"kubernetes.io/projected/d254ad6f-c371-4b41-a97d-97056891fdf6-kube-api-access-cdz5z\") pod \"ovnkube-node-v76kh\" (UID: \"d254ad6f-c371-4b41-a97d-97056891fdf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.141898 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.141873    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q265b\" (UniqueName: \"kubernetes.io/projected/3963e434-e4ab-4d73-b496-2472c7711891-kube-api-access-q265b\") pod \"node-resolver-vxf4w\" (UID: \"3963e434-e4ab-4d73-b496-2472c7711891\") " pod="openshift-dns/node-resolver-vxf4w"
Jul 13 18:49:41.142946 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.142919    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g567s\" (UniqueName: \"kubernetes.io/projected/658136e2-705b-4244-97e1-ca2913f0474f-kube-api-access-g567s\") pod \"aws-ebs-csi-driver-node-b247h\" (UID: \"658136e2-705b-4244-97e1-ca2913f0474f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:41.213099 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.213013    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j7qjq"
Jul 13 18:49:41.220637 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.220610    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gdf7f"
Jul 13 18:49:41.230164 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.230141    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bmsb4"
Jul 13 18:49:41.237122 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.237097    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:49:41.244139 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.244115    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z9pwv"
Jul 13 18:49:41.250717 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.250688    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bmcpv"
Jul 13 18:49:41.257877 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.257819    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h"
Jul 13 18:49:41.266371 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.266345    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vxf4w"
Jul 13 18:49:41.272834 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.272808    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bc97r"
Jul 13 18:49:41.624117 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:41.624088    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1baa76b_d5eb_43dc_9c73_6b4fa9f2989b.slice/crio-74c9190a7c3d46441adda565540773e13bd2541c980f047ae609c46c9d4ca189 WatchSource:0}: Error finding container 74c9190a7c3d46441adda565540773e13bd2541c980f047ae609c46c9d4ca189: Status 404 returned error can't find the container with id 74c9190a7c3d46441adda565540773e13bd2541c980f047ae609c46c9d4ca189
Jul 13 18:49:41.625598 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:41.625561    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod658136e2_705b_4244_97e1_ca2913f0474f.slice/crio-c95e4243fb827beba0d19bba93ccbde6e1b90f5cb03af48de0df9dd99ba1d176 WatchSource:0}: Error finding container c95e4243fb827beba0d19bba93ccbde6e1b90f5cb03af48de0df9dd99ba1d176: Status 404 returned error can't find the container with id c95e4243fb827beba0d19bba93ccbde6e1b90f5cb03af48de0df9dd99ba1d176
Jul 13 18:49:41.626813 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:41.626718    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeacba2f0_6acf_4ce6_a960_0f5b0c8a77d9.slice/crio-33c520e93e99e0d9ce03d09c9b662d5f5ca9c1cb7ae23d2ff422e873f270ccd6 WatchSource:0}: Error finding container 33c520e93e99e0d9ce03d09c9b662d5f5ca9c1cb7ae23d2ff422e873f270ccd6: Status 404 returned error can't find the container with id 33c520e93e99e0d9ce03d09c9b662d5f5ca9c1cb7ae23d2ff422e873f270ccd6
Jul 13 18:49:41.631253 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.631228    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret\") pod \"global-pull-secret-syncer-66pbz\" (UID: \"b1f63a2f-aa86-415d-8c72-78c96c560625\") " pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:41.631378 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.631270    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs\") pod \"network-metrics-daemon-xnfv2\" (UID: \"6ec3b9c9-8324-4d12-92b8-6baa78773fe2\") " pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:49:41.631445 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:41.631409    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:41.631445 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:41.631408    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:41.631547 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:41.631466    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs podName:6ec3b9c9-8324-4d12-92b8-6baa78773fe2 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:42.631448987 +0000 UTC m=+4.145248131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs") pod "network-metrics-daemon-xnfv2" (UID: "6ec3b9c9-8324-4d12-92b8-6baa78773fe2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:41.631596 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:41.631551    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret podName:b1f63a2f-aa86-415d-8c72-78c96c560625 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:42.63153629 +0000 UTC m=+4.145335435 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret") pod "global-pull-secret-syncer-66pbz" (UID: "b1f63a2f-aa86-415d-8c72-78c96c560625") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:41.652597 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:41.652552    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c396f9_fccb_492d_9e7e_36c4e864a2d1.slice/crio-c7fe91fb53e4c57ac0711bbd06cce786ad89adaf1c197f31acedd58dd773100c WatchSource:0}: Error finding container c7fe91fb53e4c57ac0711bbd06cce786ad89adaf1c197f31acedd58dd773100c: Status 404 returned error can't find the container with id c7fe91fb53e4c57ac0711bbd06cce786ad89adaf1c197f31acedd58dd773100c
Jul 13 18:49:41.653498 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:41.653451    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898f3564_8c07_40c7_9580_70071f193268.slice/crio-f459bab47158c213b3ca894f6b67a818c91d7fb8c54e0279dcc9a3d2feb4820e WatchSource:0}: Error finding container f459bab47158c213b3ca894f6b67a818c91d7fb8c54e0279dcc9a3d2feb4820e: Status 404 returned error can't find the container with id f459bab47158c213b3ca894f6b67a818c91d7fb8c54e0279dcc9a3d2feb4820e
Jul 13 18:49:41.731907 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.731869    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsw7c\" (UniqueName: \"kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c\") pod \"network-check-target-xrtxq\" (UID: \"6d8ac836-95a6-49cf-85ba-99e3db827c52\") " pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:49:41.732113 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:41.732089    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:49:41.732174 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:41.732121    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:49:41.732174 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:41.732137    2569 projected.go:196] Error preparing data for projected volume kube-api-access-vsw7c for pod openshift-network-diagnostics/network-check-target-xrtxq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:41.732270 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:41.732208    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c podName:6d8ac836-95a6-49cf-85ba-99e3db827c52 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:42.732186423 +0000 UTC m=+4.245985586 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vsw7c" (UniqueName: "kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c") pod "network-check-target-xrtxq" (UID: "6d8ac836-95a6-49cf-85ba-99e3db827c52") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:41.993472 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.993343    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 18:44:41 +0000 UTC" deadline="2028-01-13 22:43:01.34769679 +0000 UTC"
Jul 13 18:49:41.993472 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:41.993404    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13179h53m19.354297389s"
Jul 13 18:49:42.014726 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:42.014691    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:49:42.016952 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:42.016912    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3963e434_e4ab_4d73_b496_2472c7711891.slice/crio-cf4c556532dd2d9bcb27f6a779160b7d27c7c69b65c53fda4b3c53a238b6a7c1 WatchSource:0}: Error finding container cf4c556532dd2d9bcb27f6a779160b7d27c7c69b65c53fda4b3c53a238b6a7c1: Status 404 returned error can't find the container with id cf4c556532dd2d9bcb27f6a779160b7d27c7c69b65c53fda4b3c53a238b6a7c1
Jul 13 18:49:42.017805 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:42.017772    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd254ad6f_c371_4b41_a97d_97056891fdf6.slice/crio-cb12952080a0d2a34efaa8c96a62006acf878b0c640b6ee26402775560c4bb88 WatchSource:0}: Error finding container cb12952080a0d2a34efaa8c96a62006acf878b0c640b6ee26402775560c4bb88: Status 404 returned error can't find the container with id cb12952080a0d2a34efaa8c96a62006acf878b0c640b6ee26402775560c4bb88
Jul 13 18:49:42.018543 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:49:42.018513    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddda9308_3395_4d7b_af21_a398125bd512.slice/crio-7dd25bceff7db579fa2de978d7487ad4fa5dacff3ea2b180dd13d3e0592e59dd WatchSource:0}: Error finding container 7dd25bceff7db579fa2de978d7487ad4fa5dacff3ea2b180dd13d3e0592e59dd: Status 404 returned error can't find the container with id 7dd25bceff7db579fa2de978d7487ad4fa5dacff3ea2b180dd13d3e0592e59dd
Jul 13 18:49:42.054946 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:42.054915    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:42.055110 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:42.055028    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66pbz" podUID="b1f63a2f-aa86-415d-8c72-78c96c560625"
Jul 13 18:49:42.064774 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:42.064733    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gdf7f" event={"ID":"ddda9308-3395-4d7b-af21-a398125bd512","Type":"ContainerStarted","Data":"7dd25bceff7db579fa2de978d7487ad4fa5dacff3ea2b180dd13d3e0592e59dd"}
Jul 13 18:49:42.066053 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:42.066018    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v76kh" event={"ID":"d254ad6f-c371-4b41-a97d-97056891fdf6","Type":"ContainerStarted","Data":"cb12952080a0d2a34efaa8c96a62006acf878b0c640b6ee26402775560c4bb88"}
Jul 13 18:49:42.067410 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:42.067375    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bc97r" event={"ID":"f6c396f9-fccb-492d-9e7e-36c4e864a2d1","Type":"ContainerStarted","Data":"c7fe91fb53e4c57ac0711bbd06cce786ad89adaf1c197f31acedd58dd773100c"}
Jul 13 18:49:42.068604 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:42.068565    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bmcpv" event={"ID":"fcfca38c-2dcb-4e9d-9c13-38927d35a19f","Type":"ContainerStarted","Data":"6aab00e24dced55e01e1ff79888415c93c5f281879785999da74738ce452d227"}
Jul 13 18:49:42.070253 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:42.070223    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h" event={"ID":"658136e2-705b-4244-97e1-ca2913f0474f","Type":"ContainerStarted","Data":"c95e4243fb827beba0d19bba93ccbde6e1b90f5cb03af48de0df9dd99ba1d176"}
Jul 13 18:49:42.071553 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:42.071527    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vxf4w" event={"ID":"3963e434-e4ab-4d73-b496-2472c7711891","Type":"ContainerStarted","Data":"cf4c556532dd2d9bcb27f6a779160b7d27c7c69b65c53fda4b3c53a238b6a7c1"}
Jul 13 18:49:42.072675 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:42.072659    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bmsb4" event={"ID":"898f3564-8c07-40c7-9580-70071f193268","Type":"ContainerStarted","Data":"f459bab47158c213b3ca894f6b67a818c91d7fb8c54e0279dcc9a3d2feb4820e"}
Jul 13 18:49:42.073741 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:42.073722    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z9pwv" event={"ID":"eacba2f0-6acf-4ce6-a960-0f5b0c8a77d9","Type":"ContainerStarted","Data":"33c520e93e99e0d9ce03d09c9b662d5f5ca9c1cb7ae23d2ff422e873f270ccd6"}
Jul 13 18:49:42.074781 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:42.074763    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j7qjq" event={"ID":"e1baa76b-d5eb-43dc-9c73-6b4fa9f2989b","Type":"ContainerStarted","Data":"74c9190a7c3d46441adda565540773e13bd2541c980f047ae609c46c9d4ca189"}
Jul 13 18:49:42.640215 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:42.639845    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs\") pod \"network-metrics-daemon-xnfv2\" (UID: \"6ec3b9c9-8324-4d12-92b8-6baa78773fe2\") " pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:49:42.640421 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:42.640213    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret\") pod \"global-pull-secret-syncer-66pbz\" (UID: \"b1f63a2f-aa86-415d-8c72-78c96c560625\") " pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:42.640421 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:42.640384    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:42.640551 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:42.640457    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret podName:b1f63a2f-aa86-415d-8c72-78c96c560625 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:44.640435698 +0000 UTC m=+6.154234843 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret") pod "global-pull-secret-syncer-66pbz" (UID: "b1f63a2f-aa86-415d-8c72-78c96c560625") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:42.640911 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:42.640890    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:42.641031 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:42.640950    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs podName:6ec3b9c9-8324-4d12-92b8-6baa78773fe2 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:44.640930832 +0000 UTC m=+6.154729980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs") pod "network-metrics-daemon-xnfv2" (UID: "6ec3b9c9-8324-4d12-92b8-6baa78773fe2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:42.741544 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:42.741448    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsw7c\" (UniqueName: \"kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c\") pod \"network-check-target-xrtxq\" (UID: \"6d8ac836-95a6-49cf-85ba-99e3db827c52\") " pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:49:42.741814 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:42.741654    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:49:42.741814 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:42.741684    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:49:42.741814 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:42.741700    2569 projected.go:196] Error preparing data for projected volume kube-api-access-vsw7c for pod openshift-network-diagnostics/network-check-target-xrtxq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:42.741814 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:42.741772    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c podName:6d8ac836-95a6-49cf-85ba-99e3db827c52 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:44.741750867 +0000 UTC m=+6.255550016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vsw7c" (UniqueName: "kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c") pod "network-check-target-xrtxq" (UID: "6d8ac836-95a6-49cf-85ba-99e3db827c52") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:42.969123 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:42.969031    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:49:42.993844 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:42.993792    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 18:44:41 +0000 UTC" deadline="2028-04-30 06:20:34.884617364 +0000 UTC"
Jul 13 18:49:42.993844 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:42.993843    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15755h30m51.89078027s"
Jul 13 18:49:43.058106 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:43.058068    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:49:43.058291 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:43.058219    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrtxq" podUID="6d8ac836-95a6-49cf-85ba-99e3db827c52"
Jul 13 18:49:43.058402 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:43.058306    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:49:43.058474 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:43.058420    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnfv2" podUID="6ec3b9c9-8324-4d12-92b8-6baa78773fe2"
Jul 13 18:49:43.103057 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:43.103014    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-134.ec2.internal" event={"ID":"6d1a4d271968fb37c56f11e30e3e00f8","Type":"ContainerStarted","Data":"f77f256e960b43f3f9414c2dc9ce3ca99b08d180e1dd18daad247fff1daa10f2"}
Jul 13 18:49:43.129750 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:43.129703    2569 generic.go:356] "Generic (PLEG): container finished" podID="6c8631cd46f5ef2fe2e7fec413f6f8d7" containerID="02e1c519588969594c227de9b9a5ae8aa619c4a06b631f4a365ba5ecb31f8a99" exitCode=0
Jul 13 18:49:43.129961 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:43.129774    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-134.ec2.internal" event={"ID":"6c8631cd46f5ef2fe2e7fec413f6f8d7","Type":"ContainerDied","Data":"02e1c519588969594c227de9b9a5ae8aa619c4a06b631f4a365ba5ecb31f8a99"}
Jul 13 18:49:43.155894 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:43.155663    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-134.ec2.internal" podStartSLOduration=2.154888281 podStartE2EDuration="2.154888281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:49:43.125345578 +0000 UTC m=+4.639144750" watchObservedRunningTime="2026-07-13 18:49:43.154888281 +0000 UTC m=+4.668687452"
Jul 13 18:49:44.054832 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:44.054771    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:44.055457 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:44.054914    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66pbz" podUID="b1f63a2f-aa86-415d-8c72-78c96c560625"
Jul 13 18:49:44.169963 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:44.169907    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-134.ec2.internal" event={"ID":"6c8631cd46f5ef2fe2e7fec413f6f8d7","Type":"ContainerStarted","Data":"c67df6152a9ca09b4295c09146ac106e11c7382d4907e7168529f81ce818f2da"}
Jul 13 18:49:44.188559 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:44.188447    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-134.ec2.internal" podStartSLOduration=3.188417261 podStartE2EDuration="3.188417261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:49:44.187251432 +0000 UTC m=+5.701050601" watchObservedRunningTime="2026-07-13 18:49:44.188417261 +0000 UTC m=+5.702216426"
Jul 13 18:49:44.660588 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:44.660541    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs\") pod \"network-metrics-daemon-xnfv2\" (UID: \"6ec3b9c9-8324-4d12-92b8-6baa78773fe2\") " pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:49:44.660778 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:44.660656    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret\") pod \"global-pull-secret-syncer-66pbz\" (UID: \"b1f63a2f-aa86-415d-8c72-78c96c560625\") " pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:44.660838 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:44.660792    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:44.660909 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:44.660858    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret podName:b1f63a2f-aa86-415d-8c72-78c96c560625 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:48.660839232 +0000 UTC m=+10.174638393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret") pod "global-pull-secret-syncer-66pbz" (UID: "b1f63a2f-aa86-415d-8c72-78c96c560625") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:44.661296 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:44.661272    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:44.661446 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:44.661336    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs podName:6ec3b9c9-8324-4d12-92b8-6baa78773fe2 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:48.66131707 +0000 UTC m=+10.175116219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs") pod "network-metrics-daemon-xnfv2" (UID: "6ec3b9c9-8324-4d12-92b8-6baa78773fe2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:44.761419 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:44.761376    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsw7c\" (UniqueName: \"kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c\") pod \"network-check-target-xrtxq\" (UID: \"6d8ac836-95a6-49cf-85ba-99e3db827c52\") " pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:49:44.761627 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:44.761610    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:49:44.761705 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:44.761637    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:49:44.761705 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:44.761652    2569 projected.go:196] Error preparing data for projected volume kube-api-access-vsw7c for pod openshift-network-diagnostics/network-check-target-xrtxq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:44.761807 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:44.761720    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c podName:6d8ac836-95a6-49cf-85ba-99e3db827c52 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:48.761702362 +0000 UTC m=+10.275501529 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vsw7c" (UniqueName: "kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c") pod "network-check-target-xrtxq" (UID: "6d8ac836-95a6-49cf-85ba-99e3db827c52") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:45.057704 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:45.057610    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:49:45.058165 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:45.057827    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnfv2" podUID="6ec3b9c9-8324-4d12-92b8-6baa78773fe2"
Jul 13 18:49:45.058292 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:45.058230    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:49:45.058360 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:45.058331    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrtxq" podUID="6d8ac836-95a6-49cf-85ba-99e3db827c52"
Jul 13 18:49:46.055274 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:46.054770    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:46.055274 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:46.054900    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66pbz" podUID="b1f63a2f-aa86-415d-8c72-78c96c560625"
Jul 13 18:49:47.058882 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:47.055455    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:49:47.058882 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:47.055605    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrtxq" podUID="6d8ac836-95a6-49cf-85ba-99e3db827c52"
Jul 13 18:49:47.058882 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:47.056004    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:49:47.058882 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:47.056111    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnfv2" podUID="6ec3b9c9-8324-4d12-92b8-6baa78773fe2"
Jul 13 18:49:48.055569 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:48.055531    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:48.055778 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:48.055668    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66pbz" podUID="b1f63a2f-aa86-415d-8c72-78c96c560625"
Jul 13 18:49:48.699127 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:48.698986    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret\") pod \"global-pull-secret-syncer-66pbz\" (UID: \"b1f63a2f-aa86-415d-8c72-78c96c560625\") " pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:48.699127 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:48.699059    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs\") pod \"network-metrics-daemon-xnfv2\" (UID: \"6ec3b9c9-8324-4d12-92b8-6baa78773fe2\") " pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:49:48.699717 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:48.699165    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:48.699717 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:48.699182    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:48.699717 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:48.699246    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret podName:b1f63a2f-aa86-415d-8c72-78c96c560625 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:56.699221375 +0000 UTC m=+18.213020533 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret") pod "global-pull-secret-syncer-66pbz" (UID: "b1f63a2f-aa86-415d-8c72-78c96c560625") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:48.699717 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:48.699273    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs podName:6ec3b9c9-8324-4d12-92b8-6baa78773fe2 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:56.699259382 +0000 UTC m=+18.213058534 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs") pod "network-metrics-daemon-xnfv2" (UID: "6ec3b9c9-8324-4d12-92b8-6baa78773fe2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:48.800564 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:48.800521    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsw7c\" (UniqueName: \"kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c\") pod \"network-check-target-xrtxq\" (UID: \"6d8ac836-95a6-49cf-85ba-99e3db827c52\") " pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:49:48.800752 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:48.800720    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:49:48.800752 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:48.800748    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:49:48.800870 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:48.800763    2569 projected.go:196] Error preparing data for projected volume kube-api-access-vsw7c for pod openshift-network-diagnostics/network-check-target-xrtxq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:48.800870 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:48.800835    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c podName:6d8ac836-95a6-49cf-85ba-99e3db827c52 nodeName:}" failed. No retries permitted until 2026-07-13 18:49:56.800813444 +0000 UTC m=+18.314612614 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vsw7c" (UniqueName: "kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c") pod "network-check-target-xrtxq" (UID: "6d8ac836-95a6-49cf-85ba-99e3db827c52") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:49.055714 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:49.055603    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:49:49.055714 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:49.055628    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:49:49.055934 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:49.055731    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrtxq" podUID="6d8ac836-95a6-49cf-85ba-99e3db827c52"
Jul 13 18:49:49.056139 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:49.056102    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnfv2" podUID="6ec3b9c9-8324-4d12-92b8-6baa78773fe2"
Jul 13 18:49:50.055503 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:50.055015    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:50.055503 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:50.055146    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66pbz" podUID="b1f63a2f-aa86-415d-8c72-78c96c560625"
Jul 13 18:49:51.056329 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:51.055396    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:49:51.056329 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:51.055526    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnfv2" podUID="6ec3b9c9-8324-4d12-92b8-6baa78773fe2"
Jul 13 18:49:51.056329 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:51.056180    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:49:51.056329 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:51.056267    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrtxq" podUID="6d8ac836-95a6-49cf-85ba-99e3db827c52"
Jul 13 18:49:52.055470 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:52.055431    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:52.055635 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:52.055570    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66pbz" podUID="b1f63a2f-aa86-415d-8c72-78c96c560625"
Jul 13 18:49:53.055625 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:53.055587    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:49:53.056041 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:53.055712    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnfv2" podUID="6ec3b9c9-8324-4d12-92b8-6baa78773fe2"
Jul 13 18:49:53.056041 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:53.055778    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:49:53.056041 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:53.055896    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrtxq" podUID="6d8ac836-95a6-49cf-85ba-99e3db827c52"
Jul 13 18:49:54.055328 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:54.055233    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:54.055470 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:54.055379    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66pbz" podUID="b1f63a2f-aa86-415d-8c72-78c96c560625"
Jul 13 18:49:55.055626 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:55.055585    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:49:55.055976 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:55.055748    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnfv2" podUID="6ec3b9c9-8324-4d12-92b8-6baa78773fe2"
Jul 13 18:49:55.056211 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:55.056173    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:49:55.056368 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:55.056265    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrtxq" podUID="6d8ac836-95a6-49cf-85ba-99e3db827c52"
Jul 13 18:49:56.054767 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:56.054737    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:56.054965 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:56.054839    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66pbz" podUID="b1f63a2f-aa86-415d-8c72-78c96c560625"
Jul 13 18:49:56.764135 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:56.764097    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret\") pod \"global-pull-secret-syncer-66pbz\" (UID: \"b1f63a2f-aa86-415d-8c72-78c96c560625\") " pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:56.764567 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:56.764152    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs\") pod \"network-metrics-daemon-xnfv2\" (UID: \"6ec3b9c9-8324-4d12-92b8-6baa78773fe2\") " pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:49:56.764567 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:56.764261    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:56.764567 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:56.764262    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:56.764567 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:56.764335    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs podName:6ec3b9c9-8324-4d12-92b8-6baa78773fe2 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:12.764319687 +0000 UTC m=+34.278118830 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs") pod "network-metrics-daemon-xnfv2" (UID: "6ec3b9c9-8324-4d12-92b8-6baa78773fe2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:49:56.764567 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:56.764380    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret podName:b1f63a2f-aa86-415d-8c72-78c96c560625 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:12.76436193 +0000 UTC m=+34.278161075 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret") pod "global-pull-secret-syncer-66pbz" (UID: "b1f63a2f-aa86-415d-8c72-78c96c560625") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:49:56.865581 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:56.865546    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsw7c\" (UniqueName: \"kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c\") pod \"network-check-target-xrtxq\" (UID: \"6d8ac836-95a6-49cf-85ba-99e3db827c52\") " pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:49:56.865807 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:56.865693    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:49:56.865807 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:56.865719    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:49:56.865807 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:56.865732    2569 projected.go:196] Error preparing data for projected volume kube-api-access-vsw7c for pod openshift-network-diagnostics/network-check-target-xrtxq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:56.865807 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:56.865787    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c podName:6d8ac836-95a6-49cf-85ba-99e3db827c52 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:12.865771111 +0000 UTC m=+34.379570257 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vsw7c" (UniqueName: "kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c") pod "network-check-target-xrtxq" (UID: "6d8ac836-95a6-49cf-85ba-99e3db827c52") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:49:57.055733 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:57.055647    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:49:57.055897 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:57.055786    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnfv2" podUID="6ec3b9c9-8324-4d12-92b8-6baa78773fe2"
Jul 13 18:49:57.055897 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:57.055846    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:49:57.056011 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:57.055945    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrtxq" podUID="6d8ac836-95a6-49cf-85ba-99e3db827c52"
Jul 13 18:49:58.055523 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:58.055492    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:49:58.055926 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:58.055631    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66pbz" podUID="b1f63a2f-aa86-415d-8c72-78c96c560625"
Jul 13 18:49:59.055383 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:59.055348    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:49:59.055534 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:59.055460    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnfv2" podUID="6ec3b9c9-8324-4d12-92b8-6baa78773fe2"
Jul 13 18:49:59.055840 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:49:59.055822    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:49:59.056115 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:49:59.055914    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrtxq" podUID="6d8ac836-95a6-49cf-85ba-99e3db827c52"
Jul 13 18:50:00.054679 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:00.054638    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:50:00.054849 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:00.054758    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66pbz" podUID="b1f63a2f-aa86-415d-8c72-78c96c560625"
Jul 13 18:50:01.058899 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:01.058869    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:50:01.058899 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:01.058889    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:50:01.059433 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:01.058972    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrtxq" podUID="6d8ac836-95a6-49cf-85ba-99e3db827c52"
Jul 13 18:50:01.059433 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:01.059389    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnfv2" podUID="6ec3b9c9-8324-4d12-92b8-6baa78773fe2"
Jul 13 18:50:02.055393 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:02.055135    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:50:02.055541 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:02.055515    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66pbz" podUID="b1f63a2f-aa86-415d-8c72-78c96c560625"
Jul 13 18:50:02.211689 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:02.211652    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gdf7f" event={"ID":"ddda9308-3395-4d7b-af21-a398125bd512","Type":"ContainerStarted","Data":"6d69f5287075461409f0f4d812715ff771974cc73d97bb8a7ef503f0881d9270"}
Jul 13 18:50:02.213732 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:02.213711    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 18:50:02.214048 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:02.214029    2569 generic.go:356] "Generic (PLEG): container finished" podID="d254ad6f-c371-4b41-a97d-97056891fdf6" containerID="816ea99375d6a577f758ada54216a52df9c6da72cf5ff03b45a873baf43cad6f" exitCode=1
Jul 13 18:50:02.214127 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:02.214097    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v76kh" event={"ID":"d254ad6f-c371-4b41-a97d-97056891fdf6","Type":"ContainerDied","Data":"816ea99375d6a577f758ada54216a52df9c6da72cf5ff03b45a873baf43cad6f"}
Jul 13 18:50:02.214194 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:02.214177    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v76kh" event={"ID":"d254ad6f-c371-4b41-a97d-97056891fdf6","Type":"ContainerStarted","Data":"7df84af96e647f7c3bc62c6733f275e7799f7d07e7d44215334af0c017c03def"}
Jul 13 18:50:02.215636 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:02.215617    2569 generic.go:356] "Generic (PLEG): container finished" podID="f6c396f9-fccb-492d-9e7e-36c4e864a2d1" containerID="3d00458222f12dd956c67c6124468cf0da526e7c7bbb6b18e7e29a998a55407f" exitCode=0
Jul 13 18:50:02.215721 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:02.215676    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bc97r" event={"ID":"f6c396f9-fccb-492d-9e7e-36c4e864a2d1","Type":"ContainerDied","Data":"3d00458222f12dd956c67c6124468cf0da526e7c7bbb6b18e7e29a998a55407f"}
Jul 13 18:50:02.217339 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:02.217321    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bmcpv" event={"ID":"fcfca38c-2dcb-4e9d-9c13-38927d35a19f","Type":"ContainerStarted","Data":"f37bb51d880cc782733812ee4ff897da14e87ded458429848e29404e9f01b999"}
Jul 13 18:50:02.218919 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:02.218825    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h" event={"ID":"658136e2-705b-4244-97e1-ca2913f0474f","Type":"ContainerStarted","Data":"497cc6d271ad6387fd2de0fd97bf6d657cab6a8ab82937427c4ece31077a3a52"}
Jul 13 18:50:02.220251 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:02.220228    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vxf4w" event={"ID":"3963e434-e4ab-4d73-b496-2472c7711891","Type":"ContainerStarted","Data":"dd6dd98fb08ff952c8b3135868a5768efb83522631486bd891027f8a4779e045"}
Jul 13 18:50:02.221783 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:02.221730    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bmsb4" event={"ID":"898f3564-8c07-40c7-9580-70071f193268","Type":"ContainerStarted","Data":"567827f9681fca9a39651af23a077979bb1e2af283f923dce49f4b5a33c2ead1"}
Jul 13 18:50:02.223156 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:02.223138    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z9pwv" event={"ID":"eacba2f0-6acf-4ce6-a960-0f5b0c8a77d9","Type":"ContainerStarted","Data":"43fc7bfc8a00ed32f0647c968caedaa3645303f5798440ba1e04597f4e19cd91"}
Jul 13 18:50:02.232560 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:02.232492    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gdf7f" podStartSLOduration=3.507487342 podStartE2EDuration="23.232473285s" totalImagesPullingTime="19.724985943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:02.23182912 +0000 UTC m=+23.745628285" watchObservedRunningTime="2026-07-13 18:50:02.232473285 +0000 UTC m=+23.746272452"
Jul 13 18:50:02.258498 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:02.258440    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bmsb4" podStartSLOduration=3.111227149 podStartE2EDuration="23.258417618s" totalImagesPullingTime="20.147190469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:02.257869284 +0000 UTC m=+23.771668452" watchObservedRunningTime="2026-07-13 18:50:02.258417618 +0000 UTC m=+23.772216785"
Jul 13 18:50:02.294235 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:02.294192    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vxf4w" podStartSLOduration=3.5666048889999997 podStartE2EDuration="23.294177876s" totalImagesPullingTime="19.727572987s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:02.294082002 +0000 UTC m=+23.807881171" watchObservedRunningTime="2026-07-13 18:50:02.294177876 +0000 UTC m=+23.807977019"
Jul 13 18:50:02.294811 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:02.294778    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-z9pwv" podStartSLOduration=11.295115965 podStartE2EDuration="23.294771195s" totalImagesPullingTime="11.99965523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:02.276515967 +0000 UTC m=+23.790315127" watchObservedRunningTime="2026-07-13 18:50:02.294771195 +0000 UTC m=+23.808570360"
Jul 13 18:50:02.342575 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:02.342398    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bmcpv" podStartSLOduration=3.237905283 podStartE2EDuration="23.342380332s" totalImagesPullingTime="20.104475049s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:02.342098514 +0000 UTC m=+23.855897680" watchObservedRunningTime="2026-07-13 18:50:02.342380332 +0000 UTC m=+23.856179498"
Jul 13 18:50:03.055532 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:03.055505    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:50:03.055673 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:03.055637    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrtxq" podUID="6d8ac836-95a6-49cf-85ba-99e3db827c52"
Jul 13 18:50:03.055720 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:03.055689    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:50:03.055806 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:03.055787    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnfv2" podUID="6ec3b9c9-8324-4d12-92b8-6baa78773fe2"
Jul 13 18:50:03.227422 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:03.227384    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j7qjq" event={"ID":"e1baa76b-d5eb-43dc-9c73-6b4fa9f2989b","Type":"ContainerStarted","Data":"a04729fdd638417b3573078dc4e712b894b8c4ab26706412ee48f4416101f9de"}
Jul 13 18:50:03.230073 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:03.230053    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 18:50:03.230461 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:03.230435    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v76kh" event={"ID":"d254ad6f-c371-4b41-a97d-97056891fdf6","Type":"ContainerStarted","Data":"f65bd8c10ae6d0c2f7b844c6d14594b89fe721a98716ea870353f4572df3d17b"}
Jul 13 18:50:03.230537 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:03.230524    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v76kh" event={"ID":"d254ad6f-c371-4b41-a97d-97056891fdf6","Type":"ContainerStarted","Data":"920e9a7e6751f5777f7a5e35898313c6fed50ea041fc5b1add3f4a84620ed6c2"}
Jul 13 18:50:03.230600 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:03.230561    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v76kh" event={"ID":"d254ad6f-c371-4b41-a97d-97056891fdf6","Type":"ContainerStarted","Data":"dcf996ecc99ea0c760d40eff6c7d210950a73fc94b28da4298eba4208939ded3"}
Jul 13 18:50:03.230654 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:03.230598    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v76kh" event={"ID":"d254ad6f-c371-4b41-a97d-97056891fdf6","Type":"ContainerStarted","Data":"d154f5a7bd0adff18d282fb779cd5432b1d5708d5ca8bbf882c0e6fe0bd84903"}
Jul 13 18:50:03.246595 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:03.246547    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-j7qjq" podStartSLOduration=4.13364524 podStartE2EDuration="24.246525247s" totalImagesPullingTime="20.112880007s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:03.246039034 +0000 UTC m=+24.759838200" watchObservedRunningTime="2026-07-13 18:50:03.246525247 +0000 UTC m=+24.760324412"
Jul 13 18:50:03.492325 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:03.492268    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 13 18:50:03.980018 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:03.979891    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T18:50:03.492319535Z","UUID":"3f0acbc8-97aa-4872-b0b1-976880c7112f","Handler":null,"Name":"","Endpoint":""}
Jul 13 18:50:03.981395 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:03.981375    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 13 18:50:03.981522 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:03.981409    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 13 18:50:04.055539 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:04.055500    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:50:04.055698 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:04.055621    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66pbz" podUID="b1f63a2f-aa86-415d-8c72-78c96c560625"
Jul 13 18:50:04.235606 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:04.235521    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h" event={"ID":"658136e2-705b-4244-97e1-ca2913f0474f","Type":"ContainerStarted","Data":"1b20b38975ef9903908fb5508a43ac778328cacd0cb2b5bc07bcbd0e85ee3c65"}
Jul 13 18:50:05.055688 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:05.055467    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:50:05.055853 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:05.055512    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:50:05.055853 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:05.055819    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnfv2" podUID="6ec3b9c9-8324-4d12-92b8-6baa78773fe2"
Jul 13 18:50:05.055971 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:05.055863    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrtxq" podUID="6d8ac836-95a6-49cf-85ba-99e3db827c52"
Jul 13 18:50:05.241991 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:05.241965    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 18:50:05.242509 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:05.242431    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v76kh" event={"ID":"d254ad6f-c371-4b41-a97d-97056891fdf6","Type":"ContainerStarted","Data":"347e7a220a8085ff57c7bb2e2f3aa07aec6b3344ec0bcf895193fdb3c1470881"}
Jul 13 18:50:05.244885 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:05.244858    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h" event={"ID":"658136e2-705b-4244-97e1-ca2913f0474f","Type":"ContainerStarted","Data":"ab2b61669accc25f90883d6c825f159422ef4789d895ad9021a74da7153ae902"}
Jul 13 18:50:05.263466 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:05.263391    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b247h" podStartSLOduration=3.26462715 podStartE2EDuration="26.263372639s" totalImagesPullingTime="22.998745489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:49:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:05.263061113 +0000 UTC m=+26.776860278" watchObservedRunningTime="2026-07-13 18:50:05.263372639 +0000 UTC m=+26.777171808"
Jul 13 18:50:06.055116 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:06.055074    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:50:06.055324 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:06.055220    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66pbz" podUID="b1f63a2f-aa86-415d-8c72-78c96c560625"
Jul 13 18:50:06.130629 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:06.130592    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-z9pwv"
Jul 13 18:50:06.131179 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:06.131154    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-z9pwv"
Jul 13 18:50:06.247569 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:06.247535    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-z9pwv"
Jul 13 18:50:06.248133 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:06.247957    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z9pwv"
Jul 13 18:50:07.056143 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:07.055780    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:50:07.056143 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:07.055803    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:50:07.056143 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:07.056048    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrtxq" podUID="6d8ac836-95a6-49cf-85ba-99e3db827c52"
Jul 13 18:50:07.056143 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:07.056103    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnfv2" podUID="6ec3b9c9-8324-4d12-92b8-6baa78773fe2"
Jul 13 18:50:07.252947 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:07.252911    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 18:50:07.253503 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:07.253258    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v76kh" event={"ID":"d254ad6f-c371-4b41-a97d-97056891fdf6","Type":"ContainerStarted","Data":"9bfabc1a1bd1c041889ae603e384c2223d1d687c0dc70044f1adf4c7234be355"}
Jul 13 18:50:07.253565 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:07.253527    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:50:07.253620 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:07.253600    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:50:07.253726 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:07.253705    2569 scope.go:117] "RemoveContainer" containerID="816ea99375d6a577f758ada54216a52df9c6da72cf5ff03b45a873baf43cad6f"
Jul 13 18:50:07.255237 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:07.255213    2569 generic.go:356] "Generic (PLEG): container finished" podID="f6c396f9-fccb-492d-9e7e-36c4e864a2d1" containerID="c07fb487cd4e937dd054c9e6aa3f0f9908b1aab06361aa5609bc30cb5fa5e088" exitCode=0
Jul 13 18:50:07.255361 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:07.255274    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bc97r" event={"ID":"f6c396f9-fccb-492d-9e7e-36c4e864a2d1","Type":"ContainerDied","Data":"c07fb487cd4e937dd054c9e6aa3f0f9908b1aab06361aa5609bc30cb5fa5e088"}
Jul 13 18:50:07.271547 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:07.271518    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:50:08.055110 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:08.055077    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:50:08.055320 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:08.055184    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66pbz" podUID="b1f63a2f-aa86-415d-8c72-78c96c560625"
Jul 13 18:50:08.262092 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:08.262055    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 18:50:08.262590 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:08.262502    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v76kh" event={"ID":"d254ad6f-c371-4b41-a97d-97056891fdf6","Type":"ContainerStarted","Data":"c2340eb27c524e775114756a58926875a4c4d0b214bd7a183b448e0a958bdaea"}
Jul 13 18:50:08.262858 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:08.262829    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:50:08.285022 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:08.284996    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:50:08.295861 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:08.295802    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-v76kh" podStartSLOduration=9.507898847 podStartE2EDuration="29.295781199s" totalImagesPullingTime="19.787882352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:08.29391059 +0000 UTC m=+29.807709757" watchObservedRunningTime="2026-07-13 18:50:08.295781199 +0000 UTC m=+29.809580363"
Jul 13 18:50:08.935854 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:08.935602    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xnfv2"]
Jul 13 18:50:08.936080 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:08.935987    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:50:08.936149 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:08.936088    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnfv2" podUID="6ec3b9c9-8324-4d12-92b8-6baa78773fe2"
Jul 13 18:50:08.936389 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:08.936360    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-66pbz"]
Jul 13 18:50:08.936492 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:08.936466    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:50:08.936589 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:08.936568    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66pbz" podUID="b1f63a2f-aa86-415d-8c72-78c96c560625"
Jul 13 18:50:08.943760 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:08.943736    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xrtxq"]
Jul 13 18:50:08.943881 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:08.943833    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:50:08.943924 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:08.943903    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrtxq" podUID="6d8ac836-95a6-49cf-85ba-99e3db827c52"
Jul 13 18:50:09.266415 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:09.266383    2569 generic.go:356] "Generic (PLEG): container finished" podID="f6c396f9-fccb-492d-9e7e-36c4e864a2d1" containerID="3400c33adcce470a30d5eda1e867fa72160f8a5e896aadcb7470f645a2409e9f" exitCode=0
Jul 13 18:50:09.266938 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:09.266480    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bc97r" event={"ID":"f6c396f9-fccb-492d-9e7e-36c4e864a2d1","Type":"ContainerDied","Data":"3400c33adcce470a30d5eda1e867fa72160f8a5e896aadcb7470f645a2409e9f"}
Jul 13 18:50:10.055234 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:10.055197    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:50:10.055412 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:10.055317    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66pbz" podUID="b1f63a2f-aa86-415d-8c72-78c96c560625"
Jul 13 18:50:11.054845 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:11.054816    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:50:11.055269 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:11.054816    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:50:11.055269 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:11.054935    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnfv2" podUID="6ec3b9c9-8324-4d12-92b8-6baa78773fe2"
Jul 13 18:50:11.055269 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:11.055026    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrtxq" podUID="6d8ac836-95a6-49cf-85ba-99e3db827c52"
Jul 13 18:50:11.273666 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:11.273586    2569 generic.go:356] "Generic (PLEG): container finished" podID="f6c396f9-fccb-492d-9e7e-36c4e864a2d1" containerID="95a93048f27dc3e75a90b911d70c0b5e79f2eb91a3b1bdf6528e20020633596b" exitCode=0
Jul 13 18:50:11.273666 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:11.273642    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bc97r" event={"ID":"f6c396f9-fccb-492d-9e7e-36c4e864a2d1","Type":"ContainerDied","Data":"95a93048f27dc3e75a90b911d70c0b5e79f2eb91a3b1bdf6528e20020633596b"}
Jul 13 18:50:12.054983 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:12.054899    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:50:12.055601 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:12.055034    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66pbz" podUID="b1f63a2f-aa86-415d-8c72-78c96c560625"
Jul 13 18:50:12.783282 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:12.783241    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs\") pod \"network-metrics-daemon-xnfv2\" (UID: \"6ec3b9c9-8324-4d12-92b8-6baa78773fe2\") " pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:50:12.783491 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:12.783439    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:12.783491 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:12.783447    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret\") pod \"global-pull-secret-syncer-66pbz\" (UID: \"b1f63a2f-aa86-415d-8c72-78c96c560625\") " pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:50:12.783580 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:12.783509    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs podName:6ec3b9c9-8324-4d12-92b8-6baa78773fe2 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:44.783493379 +0000 UTC m=+66.297292535 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs") pod "network-metrics-daemon-xnfv2" (UID: "6ec3b9c9-8324-4d12-92b8-6baa78773fe2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:12.783580 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:12.783565    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:12.783676 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:12.783617    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret podName:b1f63a2f-aa86-415d-8c72-78c96c560625 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:44.783600214 +0000 UTC m=+66.297399361 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret") pod "global-pull-secret-syncer-66pbz" (UID: "b1f63a2f-aa86-415d-8c72-78c96c560625") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:12.884172 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:12.884123    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsw7c\" (UniqueName: \"kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c\") pod \"network-check-target-xrtxq\" (UID: \"6d8ac836-95a6-49cf-85ba-99e3db827c52\") " pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:50:12.884385 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:12.884266    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:50:12.884385 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:12.884284    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:50:12.884385 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:12.884307    2569 projected.go:196] Error preparing data for projected volume kube-api-access-vsw7c for pod openshift-network-diagnostics/network-check-target-xrtxq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:12.884385 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:12.884363    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c podName:6d8ac836-95a6-49cf-85ba-99e3db827c52 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:44.884348414 +0000 UTC m=+66.398147557 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vsw7c" (UniqueName: "kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c") pod "network-check-target-xrtxq" (UID: "6d8ac836-95a6-49cf-85ba-99e3db827c52") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:13.055222 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:13.055133    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:50:13.055222 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:13.055186    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:50:13.055721 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:13.055302    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnfv2" podUID="6ec3b9c9-8324-4d12-92b8-6baa78773fe2"
Jul 13 18:50:13.055721 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:13.055406    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrtxq" podUID="6d8ac836-95a6-49cf-85ba-99e3db827c52"
Jul 13 18:50:14.054785 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:14.054603    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:50:14.054925 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:14.054850    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66pbz" podUID="b1f63a2f-aa86-415d-8c72-78c96c560625"
Jul 13 18:50:15.054900 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:15.054859    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:50:15.055394 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:15.055000    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnfv2" podUID="6ec3b9c9-8324-4d12-92b8-6baa78773fe2"
Jul 13 18:50:15.055394 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:15.055055    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:50:15.055394 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:15.055166    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrtxq" podUID="6d8ac836-95a6-49cf-85ba-99e3db827c52"
Jul 13 18:50:15.783550 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:15.783525    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-134.ec2.internal" event="NodeReady"
Jul 13 18:50:15.783693 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:15.783653    2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 13 18:50:15.838685 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:15.838659    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xzzrg"]
Jul 13 18:50:15.842080 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:15.842049    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:50:15.843017 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:15.842993    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9qzhl"]
Jul 13 18:50:15.845634 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:15.845591    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 13 18:50:15.845783 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:15.845763    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-88cff\""
Jul 13 18:50:15.845931 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:15.845907    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 13 18:50:15.846442 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:15.846424    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9qzhl"
Jul 13 18:50:15.849020 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:15.849000    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 13 18:50:15.849112 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:15.849001    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 13 18:50:15.849112 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:15.849003    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 13 18:50:15.849532 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:15.849514    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bq5zm\""
Jul 13 18:50:15.853721 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:15.853696    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xzzrg"]
Jul 13 18:50:15.857164 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:15.857139    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9qzhl"]
Jul 13 18:50:16.005707 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.005603    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kd8s\" (UniqueName: \"kubernetes.io/projected/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-kube-api-access-5kd8s\") pod \"dns-default-xzzrg\" (UID: \"c4789bc2-8df6-4746-b737-e6f4fd4a9f99\") " pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:50:16.005707 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.005658    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls\") pod \"dns-default-xzzrg\" (UID: \"c4789bc2-8df6-4746-b737-e6f4fd4a9f99\") " pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:50:16.005707 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.005689    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xg6x\" (UniqueName: \"kubernetes.io/projected/502189f5-f517-4aa2-b359-07b8c5be3b9a-kube-api-access-8xg6x\") pod \"ingress-canary-9qzhl\" (UID: \"502189f5-f517-4aa2-b359-07b8c5be3b9a\") " pod="openshift-ingress-canary/ingress-canary-9qzhl"
Jul 13 18:50:16.005707 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.005713    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/502189f5-f517-4aa2-b359-07b8c5be3b9a-cert\") pod \"ingress-canary-9qzhl\" (UID: \"502189f5-f517-4aa2-b359-07b8c5be3b9a\") " pod="openshift-ingress-canary/ingress-canary-9qzhl"
Jul 13 18:50:16.005956 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.005769    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-config-volume\") pod \"dns-default-xzzrg\" (UID: \"c4789bc2-8df6-4746-b737-e6f4fd4a9f99\") " pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:50:16.005956 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.005791    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-tmp-dir\") pod \"dns-default-xzzrg\" (UID: \"c4789bc2-8df6-4746-b737-e6f4fd4a9f99\") " pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:50:16.055092 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.055051    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:50:16.058244 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.058222    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 18:50:16.106760 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.106717    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-config-volume\") pod \"dns-default-xzzrg\" (UID: \"c4789bc2-8df6-4746-b737-e6f4fd4a9f99\") " pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:50:16.106760 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.106758    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-tmp-dir\") pod \"dns-default-xzzrg\" (UID: \"c4789bc2-8df6-4746-b737-e6f4fd4a9f99\") " pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:50:16.106961 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.106913    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kd8s\" (UniqueName: \"kubernetes.io/projected/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-kube-api-access-5kd8s\") pod \"dns-default-xzzrg\" (UID: \"c4789bc2-8df6-4746-b737-e6f4fd4a9f99\") " pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:50:16.106961 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.106940    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls\") pod \"dns-default-xzzrg\" (UID: \"c4789bc2-8df6-4746-b737-e6f4fd4a9f99\") " pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:50:16.107065 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.106961    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xg6x\" (UniqueName: \"kubernetes.io/projected/502189f5-f517-4aa2-b359-07b8c5be3b9a-kube-api-access-8xg6x\") pod \"ingress-canary-9qzhl\" (UID: \"502189f5-f517-4aa2-b359-07b8c5be3b9a\") " pod="openshift-ingress-canary/ingress-canary-9qzhl"
Jul 13 18:50:16.107065 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.106980    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/502189f5-f517-4aa2-b359-07b8c5be3b9a-cert\") pod \"ingress-canary-9qzhl\" (UID: \"502189f5-f517-4aa2-b359-07b8c5be3b9a\") " pod="openshift-ingress-canary/ingress-canary-9qzhl"
Jul 13 18:50:16.107065 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.107036    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-tmp-dir\") pod \"dns-default-xzzrg\" (UID: \"c4789bc2-8df6-4746-b737-e6f4fd4a9f99\") " pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:50:16.107192 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:16.107124    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:50:16.107230 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:16.107201    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls podName:c4789bc2-8df6-4746-b737-e6f4fd4a9f99 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:16.607182244 +0000 UTC m=+38.120981388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls") pod "dns-default-xzzrg" (UID: "c4789bc2-8df6-4746-b737-e6f4fd4a9f99") : secret "dns-default-metrics-tls" not found
Jul 13 18:50:16.107397 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.107376    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-config-volume\") pod \"dns-default-xzzrg\" (UID: \"c4789bc2-8df6-4746-b737-e6f4fd4a9f99\") " pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:50:16.111341 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.111321    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/502189f5-f517-4aa2-b359-07b8c5be3b9a-cert\") pod \"ingress-canary-9qzhl\" (UID: \"502189f5-f517-4aa2-b359-07b8c5be3b9a\") " pod="openshift-ingress-canary/ingress-canary-9qzhl"
Jul 13 18:50:16.116444 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.116417    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kd8s\" (UniqueName: \"kubernetes.io/projected/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-kube-api-access-5kd8s\") pod \"dns-default-xzzrg\" (UID: \"c4789bc2-8df6-4746-b737-e6f4fd4a9f99\") " pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:50:16.116592 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.116573    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xg6x\" (UniqueName: \"kubernetes.io/projected/502189f5-f517-4aa2-b359-07b8c5be3b9a-kube-api-access-8xg6x\") pod \"ingress-canary-9qzhl\" (UID: \"502189f5-f517-4aa2-b359-07b8c5be3b9a\") " pod="openshift-ingress-canary/ingress-canary-9qzhl"
Jul 13 18:50:16.225471 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.225434    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9qzhl"
Jul 13 18:50:16.289971 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.289943    2569 generic.go:356] "Generic (PLEG): container finished" podID="f6c396f9-fccb-492d-9e7e-36c4e864a2d1" containerID="95b47aee7ad4dd4195264f05fe92992b35350cbecdcba8dc19db063cb38517fe" exitCode=0
Jul 13 18:50:16.290116 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.290000    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bc97r" event={"ID":"f6c396f9-fccb-492d-9e7e-36c4e864a2d1","Type":"ContainerDied","Data":"95b47aee7ad4dd4195264f05fe92992b35350cbecdcba8dc19db063cb38517fe"}
Jul 13 18:50:16.384670 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.384642    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9qzhl"]
Jul 13 18:50:16.387462 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:50:16.387426    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502189f5_f517_4aa2_b359_07b8c5be3b9a.slice/crio-5216e3adb81af0f4b5c41a0a04b514bba202ff11f0e2fd5d62a7fb1ba4be082c WatchSource:0}: Error finding container 5216e3adb81af0f4b5c41a0a04b514bba202ff11f0e2fd5d62a7fb1ba4be082c: Status 404 returned error can't find the container with id 5216e3adb81af0f4b5c41a0a04b514bba202ff11f0e2fd5d62a7fb1ba4be082c
Jul 13 18:50:16.609707 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:16.609466    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls\") pod \"dns-default-xzzrg\" (UID: \"c4789bc2-8df6-4746-b737-e6f4fd4a9f99\") " pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:50:16.609707 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:16.609610    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:50:16.609934 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:16.609750    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls podName:c4789bc2-8df6-4746-b737-e6f4fd4a9f99 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:17.609731855 +0000 UTC m=+39.123531000 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls") pod "dns-default-xzzrg" (UID: "c4789bc2-8df6-4746-b737-e6f4fd4a9f99") : secret "dns-default-metrics-tls" not found
Jul 13 18:50:17.055711 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:17.055626    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:50:17.056143 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:17.055626    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:50:17.058845 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:17.058822    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 13 18:50:17.058977 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:17.058880    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 13 18:50:17.059965 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:17.059941    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 18:50:17.059965 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:17.059963    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dz6tv\""
Jul 13 18:50:17.060118 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:17.059973    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gsq9l\""
Jul 13 18:50:17.295815 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:17.295783    2569 generic.go:356] "Generic (PLEG): container finished" podID="f6c396f9-fccb-492d-9e7e-36c4e864a2d1" containerID="c73001880c215dd517aa434de3773b58c847e9aed03bc73752b8d521ad6ebc9a" exitCode=0
Jul 13 18:50:17.296004 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:17.295861    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bc97r" event={"ID":"f6c396f9-fccb-492d-9e7e-36c4e864a2d1","Type":"ContainerDied","Data":"c73001880c215dd517aa434de3773b58c847e9aed03bc73752b8d521ad6ebc9a"}
Jul 13 18:50:17.297373 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:17.297344    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9qzhl" event={"ID":"502189f5-f517-4aa2-b359-07b8c5be3b9a","Type":"ContainerStarted","Data":"5216e3adb81af0f4b5c41a0a04b514bba202ff11f0e2fd5d62a7fb1ba4be082c"}
Jul 13 18:50:17.617946 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:17.617915    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls\") pod \"dns-default-xzzrg\" (UID: \"c4789bc2-8df6-4746-b737-e6f4fd4a9f99\") " pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:50:17.618145 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:17.618092    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:50:17.618194 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:17.618168    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls podName:c4789bc2-8df6-4746-b737-e6f4fd4a9f99 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:19.61815139 +0000 UTC m=+41.131950553 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls") pod "dns-default-xzzrg" (UID: "c4789bc2-8df6-4746-b737-e6f4fd4a9f99") : secret "dns-default-metrics-tls" not found
Jul 13 18:50:18.303444 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:18.303398    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bc97r" event={"ID":"f6c396f9-fccb-492d-9e7e-36c4e864a2d1","Type":"ContainerStarted","Data":"62155d65e346b792c665db8b4987b55bf94cc88fc58847ce42711f4651a3643c"}
Jul 13 18:50:18.330227 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:18.329768    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bc97r" podStartSLOduration=7.179446363 podStartE2EDuration="39.329738322s" totalImagesPullingTime="32.150291959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:49:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:18.327955613 +0000 UTC m=+39.841754778" watchObservedRunningTime="2026-07-13 18:50:18.329738322 +0000 UTC m=+39.843537490"
Jul 13 18:50:19.307409 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:19.307375    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9qzhl" event={"ID":"502189f5-f517-4aa2-b359-07b8c5be3b9a","Type":"ContainerStarted","Data":"fad0236dc0defa3bd14bc6b03702a6cbd4867f38aeeaa372be1246a5b0cc9812"}
Jul 13 18:50:19.325886 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:19.325828    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9qzhl" podStartSLOduration=1.6160652039999999 podStartE2EDuration="4.32581122s" totalImagesPullingTime="2.709746016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:19.324787462 +0000 UTC m=+40.838586629" watchObservedRunningTime="2026-07-13 18:50:19.32581122 +0000 UTC m=+40.839610386"
Jul 13 18:50:19.636067 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:19.636025    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls\") pod \"dns-default-xzzrg\" (UID: \"c4789bc2-8df6-4746-b737-e6f4fd4a9f99\") " pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:50:19.636252 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:19.636154    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:50:19.636252 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:19.636206    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls podName:c4789bc2-8df6-4746-b737-e6f4fd4a9f99 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:23.636193159 +0000 UTC m=+45.149992302 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls") pod "dns-default-xzzrg" (UID: "c4789bc2-8df6-4746-b737-e6f4fd4a9f99") : secret "dns-default-metrics-tls" not found
Jul 13 18:50:23.668394 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:23.668353    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls\") pod \"dns-default-xzzrg\" (UID: \"c4789bc2-8df6-4746-b737-e6f4fd4a9f99\") " pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:50:23.668772 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:23.668483    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:50:23.668772 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:23.668534    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls podName:c4789bc2-8df6-4746-b737-e6f4fd4a9f99 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:31.668521661 +0000 UTC m=+53.182320805 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls") pod "dns-default-xzzrg" (UID: "c4789bc2-8df6-4746-b737-e6f4fd4a9f99") : secret "dns-default-metrics-tls" not found
Jul 13 18:50:31.726473 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:31.726429    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls\") pod \"dns-default-xzzrg\" (UID: \"c4789bc2-8df6-4746-b737-e6f4fd4a9f99\") " pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:50:31.727001 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:31.726552    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:50:31.727001 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:31.726606    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls podName:c4789bc2-8df6-4746-b737-e6f4fd4a9f99 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:47.726591869 +0000 UTC m=+69.240391013 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls") pod "dns-default-xzzrg" (UID: "c4789bc2-8df6-4746-b737-e6f4fd4a9f99") : secret "dns-default-metrics-tls" not found
Jul 13 18:50:40.279528 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:40.279497    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v76kh"
Jul 13 18:50:44.814902 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:44.814863    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret\") pod \"global-pull-secret-syncer-66pbz\" (UID: \"b1f63a2f-aa86-415d-8c72-78c96c560625\") " pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:50:44.815398 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:44.814917    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs\") pod \"network-metrics-daemon-xnfv2\" (UID: \"6ec3b9c9-8324-4d12-92b8-6baa78773fe2\") " pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:50:44.817862 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:44.817841    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 18:50:44.817959 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:44.817940    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 18:50:44.825693 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:44.825666    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 13 18:50:44.825760 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:44.825749    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs podName:6ec3b9c9-8324-4d12-92b8-6baa78773fe2 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:48.825730978 +0000 UTC m=+130.339530121 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs") pod "network-metrics-daemon-xnfv2" (UID: "6ec3b9c9-8324-4d12-92b8-6baa78773fe2") : secret "metrics-daemon-secret" not found
Jul 13 18:50:44.829172 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:44.829141    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1f63a2f-aa86-415d-8c72-78c96c560625-original-pull-secret\") pod \"global-pull-secret-syncer-66pbz\" (UID: \"b1f63a2f-aa86-415d-8c72-78c96c560625\") " pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:50:44.862858 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:44.862817    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66pbz"
Jul 13 18:50:44.915567 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:44.915532    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsw7c\" (UniqueName: \"kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c\") pod \"network-check-target-xrtxq\" (UID: \"6d8ac836-95a6-49cf-85ba-99e3db827c52\") " pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:50:44.918824 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:44.918771    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 13 18:50:44.928659 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:44.928585    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 13 18:50:44.938822 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:44.938793    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsw7c\" (UniqueName: \"kubernetes.io/projected/6d8ac836-95a6-49cf-85ba-99e3db827c52-kube-api-access-vsw7c\") pod \"network-check-target-xrtxq\" (UID: \"6d8ac836-95a6-49cf-85ba-99e3db827c52\") " pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:50:44.976762 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:44.976723    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dz6tv\""
Jul 13 18:50:44.984061 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:44.984038    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:50:44.994313 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:44.994270    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-66pbz"]
Jul 13 18:50:44.998866 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:50:44.998837    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f63a2f_aa86_415d_8c72_78c96c560625.slice/crio-ce1baac7ea8f4436d6fc1c3768225cfe229c5f5d94a91a053ab289081064a174 WatchSource:0}: Error finding container ce1baac7ea8f4436d6fc1c3768225cfe229c5f5d94a91a053ab289081064a174: Status 404 returned error can't find the container with id ce1baac7ea8f4436d6fc1c3768225cfe229c5f5d94a91a053ab289081064a174
Jul 13 18:50:45.108943 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:45.108895    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xrtxq"]
Jul 13 18:50:45.113222 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:50:45.113197    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8ac836_95a6_49cf_85ba_99e3db827c52.slice/crio-1f972a77baf29b371ae0452be59407b7ed5a322fcfad8266fce6db1596cdf929 WatchSource:0}: Error finding container 1f972a77baf29b371ae0452be59407b7ed5a322fcfad8266fce6db1596cdf929: Status 404 returned error can't find the container with id 1f972a77baf29b371ae0452be59407b7ed5a322fcfad8266fce6db1596cdf929
Jul 13 18:50:45.372219 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:45.372127    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-66pbz" event={"ID":"b1f63a2f-aa86-415d-8c72-78c96c560625","Type":"ContainerStarted","Data":"ce1baac7ea8f4436d6fc1c3768225cfe229c5f5d94a91a053ab289081064a174"}
Jul 13 18:50:45.373083 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:45.373057    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xrtxq" event={"ID":"6d8ac836-95a6-49cf-85ba-99e3db827c52","Type":"ContainerStarted","Data":"1f972a77baf29b371ae0452be59407b7ed5a322fcfad8266fce6db1596cdf929"}
Jul 13 18:50:47.736939 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:47.736899    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls\") pod \"dns-default-xzzrg\" (UID: \"c4789bc2-8df6-4746-b737-e6f4fd4a9f99\") " pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:50:47.737434 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:47.737029    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:50:47.737434 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:47.737086    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls podName:c4789bc2-8df6-4746-b737-e6f4fd4a9f99 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:19.737072449 +0000 UTC m=+101.250871592 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls") pod "dns-default-xzzrg" (UID: "c4789bc2-8df6-4746-b737-e6f4fd4a9f99") : secret "dns-default-metrics-tls" not found
Jul 13 18:50:50.388249 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:50.388209    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-66pbz" event={"ID":"b1f63a2f-aa86-415d-8c72-78c96c560625","Type":"ContainerStarted","Data":"eee09a8f2ea59123f024af5e16c5719c6862ead384910495d748cbf22f53a273"}
Jul 13 18:50:50.389611 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:50.389590    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xrtxq" event={"ID":"6d8ac836-95a6-49cf-85ba-99e3db827c52","Type":"ContainerStarted","Data":"d1511b6571ae63792ebec9990fca6eae3611bd52bce6fe83d6e421574b8c5e13"}
Jul 13 18:50:50.389741 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:50.389723    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:50:50.405652 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:50.405603    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-66pbz" podStartSLOduration=66.415860787 podStartE2EDuration="1m11.405587629s" totalImagesPullingTime="4.989726842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:50.404919001 +0000 UTC m=+71.918718167" watchObservedRunningTime="2026-07-13 18:50:50.405587629 +0000 UTC m=+71.919386825"
Jul 13 18:50:50.421247 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:50.421192    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xrtxq" podStartSLOduration=66.55721096 podStartE2EDuration="1m11.421172142s" totalImagesPullingTime="4.863961182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:50.420074172 +0000 UTC m=+71.933873337" watchObservedRunningTime="2026-07-13 18:50:50.421172142 +0000 UTC m=+71.934971309"
Jul 13 18:50:54.888388 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:54.888356    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-9pt94"]
Jul 13 18:50:54.890809 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:54.890793    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-9pt94"
Jul 13 18:50:54.903250 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:54.903216    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 13 18:50:54.903415 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:54.903224    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 13 18:50:54.903415 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:54.903224    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 13 18:50:54.903415 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:54.903224    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 13 18:50:54.904450 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:54.904432    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qlnvf\""
Jul 13 18:50:54.912111 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:54.912084    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-9pt94"]
Jul 13 18:50:54.988228 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:54.988188    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7bd176fb-f16d-4c0b-99ac-ef0bb25b1390-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-9pt94\" (UID: \"7bd176fb-f16d-4c0b-99ac-ef0bb25b1390\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-9pt94"
Jul 13 18:50:54.988465 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:54.988254    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7bd176fb-f16d-4c0b-99ac-ef0bb25b1390-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-9pt94\" (UID: \"7bd176fb-f16d-4c0b-99ac-ef0bb25b1390\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-9pt94"
Jul 13 18:50:54.988465 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:54.988276    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgr5t\" (UniqueName: \"kubernetes.io/projected/7bd176fb-f16d-4c0b-99ac-ef0bb25b1390-kube-api-access-lgr5t\") pod \"kube-storage-version-migrator-operator-799cb7f958-9pt94\" (UID: \"7bd176fb-f16d-4c0b-99ac-ef0bb25b1390\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-9pt94"
Jul 13 18:50:55.009487 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.009454    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-tbqhd"]
Jul 13 18:50:55.011766 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.011747    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd"
Jul 13 18:50:55.015731 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.015705    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 13 18:50:55.017514 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.017480    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-67dpg\""
Jul 13 18:50:55.017791 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.017654    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 13 18:50:55.018008 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.017991    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 13 18:50:55.018196 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.018180    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 13 18:50:55.023227 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.023211    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 13 18:50:55.033612 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.033591    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-tbqhd"]
Jul 13 18:50:55.088770 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.088725    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7bd176fb-f16d-4c0b-99ac-ef0bb25b1390-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-9pt94\" (UID: \"7bd176fb-f16d-4c0b-99ac-ef0bb25b1390\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-9pt94"
Jul 13 18:50:55.088945 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.088774    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8977c40b-f5c4-4aea-a441-6611214b30f7-tmp\") pod \"insights-operator-68d9dd94cf-tbqhd\" (UID: \"8977c40b-f5c4-4aea-a441-6611214b30f7\") " pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd"
Jul 13 18:50:55.088945 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.088831    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8977c40b-f5c4-4aea-a441-6611214b30f7-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-tbqhd\" (UID: \"8977c40b-f5c4-4aea-a441-6611214b30f7\") " pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd"
Jul 13 18:50:55.088945 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.088858    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8977c40b-f5c4-4aea-a441-6611214b30f7-serving-cert\") pod \"insights-operator-68d9dd94cf-tbqhd\" (UID: \"8977c40b-f5c4-4aea-a441-6611214b30f7\") " pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd"
Jul 13 18:50:55.089048 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.088958    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7bd176fb-f16d-4c0b-99ac-ef0bb25b1390-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-9pt94\" (UID: \"7bd176fb-f16d-4c0b-99ac-ef0bb25b1390\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-9pt94"
Jul 13 18:50:55.089048 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.088996    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8977c40b-f5c4-4aea-a441-6611214b30f7-snapshots\") pod \"insights-operator-68d9dd94cf-tbqhd\" (UID: \"8977c40b-f5c4-4aea-a441-6611214b30f7\") " pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd"
Jul 13 18:50:55.089048 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.089021    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgr5t\" (UniqueName: \"kubernetes.io/projected/7bd176fb-f16d-4c0b-99ac-ef0bb25b1390-kube-api-access-lgr5t\") pod \"kube-storage-version-migrator-operator-799cb7f958-9pt94\" (UID: \"7bd176fb-f16d-4c0b-99ac-ef0bb25b1390\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-9pt94"
Jul 13 18:50:55.089155 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.089135    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8977c40b-f5c4-4aea-a441-6611214b30f7-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-tbqhd\" (UID: \"8977c40b-f5c4-4aea-a441-6611214b30f7\") " pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd"
Jul 13 18:50:55.089189 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.089175    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nkb9\" (UniqueName: \"kubernetes.io/projected/8977c40b-f5c4-4aea-a441-6611214b30f7-kube-api-access-2nkb9\") pod \"insights-operator-68d9dd94cf-tbqhd\" (UID: \"8977c40b-f5c4-4aea-a441-6611214b30f7\") " pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd"
Jul 13 18:50:55.089447 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.089428    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7bd176fb-f16d-4c0b-99ac-ef0bb25b1390-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-9pt94\" (UID: \"7bd176fb-f16d-4c0b-99ac-ef0bb25b1390\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-9pt94"
Jul 13 18:50:55.091335 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.091320    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7bd176fb-f16d-4c0b-99ac-ef0bb25b1390-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-9pt94\" (UID: \"7bd176fb-f16d-4c0b-99ac-ef0bb25b1390\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-9pt94"
Jul 13 18:50:55.098118 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.098092    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-s92dx"]
Jul 13 18:50:55.100506 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.100491    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-s92dx"
Jul 13 18:50:55.106163 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:55.106129    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"serving-cert\" is forbidden: User \"system:node:ip-10-0-139-134.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-console-operator\": no relationship found between node 'ip-10-0-139-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" type="*v1.Secret"
Jul 13 18:50:55.106389 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:55.106129    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"trusted-ca\" is forbidden: User \"system:node:ip-10-0-139-134.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console-operator\": no relationship found between node 'ip-10-0-139-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" type="*v1.ConfigMap"
Jul 13 18:50:55.106389 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:55.106237    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"console-operator-7866497494-s92dx\" is forbidden: User \"system:node:ip-10-0-139-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-console-operator\": no relationship found between node 'ip-10-0-139-134.ec2.internal' and this object" podUID="a769d459-df42-47dc-8cfd-08a32d48bcea" pod="openshift-console-operator/console-operator-7866497494-s92dx"
Jul 13 18:50:55.106389 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:55.106259    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-139-134.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console-operator\": no relationship found between node 'ip-10-0-139-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 13 18:50:55.106389 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:55.106232    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-139-134.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console-operator\": no relationship found between node 'ip-10-0-139-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 13 18:50:55.106389 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:55.106322    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"console-operator-dockercfg-48thm\" is forbidden: User \"system:node:ip-10-0-139-134.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-console-operator\": no relationship found between node 'ip-10-0-139-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-48thm\"" type="*v1.Secret"
Jul 13 18:50:55.106765 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:55.106280    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"console-operator-config\" is forbidden: User \"system:node:ip-10-0-139-134.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console-operator\": no relationship found between node 'ip-10-0-139-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" type="*v1.ConfigMap"
Jul 13 18:50:55.106765 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.106493    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-586b57644b-q24lm"]
Jul 13 18:50:55.108846 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.108830    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:50:55.113872 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:55.113838    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-139-134.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 13 18:50:55.115113 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:55.115082    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"router-default-586b57644b-q24lm\" is forbidden: User \"system:node:ip-10-0-139-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-134.ec2.internal' and this object" podUID="4fcfa1f0-d0ea-488a-a414-d09b8085aae5" pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:50:55.115113 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:55.115100    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"router-dockercfg-7jj6d\" is forbidden: User \"system:node:ip-10-0-139-134.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7jj6d\"" type="*v1.Secret"
Jul 13 18:50:55.115245 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:55.115089    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"router-stats-default\" is forbidden: User \"system:node:ip-10-0-139-134.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" type="*v1.Secret"
Jul 13 18:50:55.115245 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:55.115180    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"service-ca-bundle\" is forbidden: User \"system:node:ip-10-0-139-134.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" type="*v1.ConfigMap"
Jul 13 18:50:55.115557 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:55.115539    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-139-134.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 13 18:50:55.115657 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:55.115641    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"router-metrics-certs-default\" is forbidden: User \"system:node:ip-10-0-139-134.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" type="*v1.Secret"
Jul 13 18:50:55.115813 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:55.115795    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-ingress-cert\" is forbidden: User \"system:node:ip-10-0-139-134.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" type="*v1.Secret"
Jul 13 18:50:55.118816 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:55.118791    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"console-operator-7866497494-s92dx\" is forbidden: User \"system:node:ip-10-0-139-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-console-operator\": no relationship found between node 'ip-10-0-139-134.ec2.internal' and this object" podUID="a769d459-df42-47dc-8cfd-08a32d48bcea" pod="openshift-console-operator/console-operator-7866497494-s92dx"
Jul 13 18:50:55.124430 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.124408    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64f848497f-8c2qx"]
Jul 13 18:50:55.125308 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.125270    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgr5t\" (UniqueName: \"kubernetes.io/projected/7bd176fb-f16d-4c0b-99ac-ef0bb25b1390-kube-api-access-lgr5t\") pod \"kube-storage-version-migrator-operator-799cb7f958-9pt94\" (UID: \"7bd176fb-f16d-4c0b-99ac-ef0bb25b1390\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-9pt94"
Jul 13 18:50:55.127119 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.127104    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.136854 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.136833    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 13 18:50:55.136986 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.136971    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 13 18:50:55.138676 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.138628    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 13 18:50:55.138865 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.138850    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p56b4\""
Jul 13 18:50:55.152574 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.152548    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 13 18:50:55.162028 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.162001    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-586b57644b-q24lm"]
Jul 13 18:50:55.165858 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.165835    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-s92dx"]
Jul 13 18:50:55.168162 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.168142    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64f848497f-8c2qx"]
Jul 13 18:50:55.189571 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.189536    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a769d459-df42-47dc-8cfd-08a32d48bcea-trusted-ca\") pod \"console-operator-7866497494-s92dx\" (UID: \"a769d459-df42-47dc-8cfd-08a32d48bcea\") " pod="openshift-console-operator/console-operator-7866497494-s92dx"
Jul 13 18:50:55.189571 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.189571    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73a82b38-5c25-40f0-9849-4b0be568ae02-image-registry-private-configuration\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.189780 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.189593    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8977c40b-f5c4-4aea-a441-6611214b30f7-snapshots\") pod \"insights-operator-68d9dd94cf-tbqhd\" (UID: \"8977c40b-f5c4-4aea-a441-6611214b30f7\") " pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd"
Jul 13 18:50:55.189780 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.189622    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7zxw\" (UniqueName: \"kubernetes.io/projected/a769d459-df42-47dc-8cfd-08a32d48bcea-kube-api-access-z7zxw\") pod \"console-operator-7866497494-s92dx\" (UID: \"a769d459-df42-47dc-8cfd-08a32d48bcea\") " pod="openshift-console-operator/console-operator-7866497494-s92dx"
Jul 13 18:50:55.189780 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.189646    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73a82b38-5c25-40f0-9849-4b0be568ae02-installation-pull-secrets\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.189780 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.189663    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-service-ca-bundle\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:50:55.189780 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.189678    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-tls\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.189780 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.189700    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nkb9\" (UniqueName: \"kubernetes.io/projected/8977c40b-f5c4-4aea-a441-6611214b30f7-kube-api-access-2nkb9\") pod \"insights-operator-68d9dd94cf-tbqhd\" (UID: \"8977c40b-f5c4-4aea-a441-6611214b30f7\") " pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd"
Jul 13 18:50:55.189780 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.189723    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-bound-sa-token\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.190073 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.189816    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a769d459-df42-47dc-8cfd-08a32d48bcea-serving-cert\") pod \"console-operator-7866497494-s92dx\" (UID: \"a769d459-df42-47dc-8cfd-08a32d48bcea\") " pod="openshift-console-operator/console-operator-7866497494-s92dx"
Jul 13 18:50:55.190073 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.189849    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8977c40b-f5c4-4aea-a441-6611214b30f7-serving-cert\") pod \"insights-operator-68d9dd94cf-tbqhd\" (UID: \"8977c40b-f5c4-4aea-a441-6611214b30f7\") " pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd"
Jul 13 18:50:55.190073 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.189873    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-metrics-certs\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:50:55.190073 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.189897    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a769d459-df42-47dc-8cfd-08a32d48bcea-config\") pod \"console-operator-7866497494-s92dx\" (UID: \"a769d459-df42-47dc-8cfd-08a32d48bcea\") " pod="openshift-console-operator/console-operator-7866497494-s92dx"
Jul 13 18:50:55.190073 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.189911    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73a82b38-5c25-40f0-9849-4b0be568ae02-ca-trust-extracted\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.190073 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.189935    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-certificates\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.190073 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.189956    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhklf\" (UniqueName: \"kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-kube-api-access-lhklf\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.190073 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.190036    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr5kv\" (UniqueName: \"kubernetes.io/projected/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-kube-api-access-kr5kv\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:50:55.190073 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.190073    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8977c40b-f5c4-4aea-a441-6611214b30f7-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-tbqhd\" (UID: \"8977c40b-f5c4-4aea-a441-6611214b30f7\") " pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd"
Jul 13 18:50:55.190521 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.190107    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-stats-auth\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:50:55.190521 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.190131    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73a82b38-5c25-40f0-9849-4b0be568ae02-trusted-ca\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.190521 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.190173    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8977c40b-f5c4-4aea-a441-6611214b30f7-tmp\") pod \"insights-operator-68d9dd94cf-tbqhd\" (UID: \"8977c40b-f5c4-4aea-a441-6611214b30f7\") " pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd"
Jul 13 18:50:55.190521 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.190205    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8977c40b-f5c4-4aea-a441-6611214b30f7-snapshots\") pod \"insights-operator-68d9dd94cf-tbqhd\" (UID: \"8977c40b-f5c4-4aea-a441-6611214b30f7\") " pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd"
Jul 13 18:50:55.190521 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.190204    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8977c40b-f5c4-4aea-a441-6611214b30f7-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-tbqhd\" (UID: \"8977c40b-f5c4-4aea-a441-6611214b30f7\") " pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd"
Jul 13 18:50:55.190521 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.190248    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-default-certificate\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:50:55.190521 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.190437    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8977c40b-f5c4-4aea-a441-6611214b30f7-tmp\") pod \"insights-operator-68d9dd94cf-tbqhd\" (UID: \"8977c40b-f5c4-4aea-a441-6611214b30f7\") " pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd"
Jul 13 18:50:55.190844 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.190677    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8977c40b-f5c4-4aea-a441-6611214b30f7-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-tbqhd\" (UID: \"8977c40b-f5c4-4aea-a441-6611214b30f7\") " pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd"
Jul 13 18:50:55.191079 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.191058    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8977c40b-f5c4-4aea-a441-6611214b30f7-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-tbqhd\" (UID: \"8977c40b-f5c4-4aea-a441-6611214b30f7\") " pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd"
Jul 13 18:50:55.192401 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.192385    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8977c40b-f5c4-4aea-a441-6611214b30f7-serving-cert\") pod \"insights-operator-68d9dd94cf-tbqhd\" (UID: \"8977c40b-f5c4-4aea-a441-6611214b30f7\") " pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd"
Jul 13 18:50:55.199027 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.199008    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-9pt94"
Jul 13 18:50:55.201579 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.201554    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nkb9\" (UniqueName: \"kubernetes.io/projected/8977c40b-f5c4-4aea-a441-6611214b30f7-kube-api-access-2nkb9\") pod \"insights-operator-68d9dd94cf-tbqhd\" (UID: \"8977c40b-f5c4-4aea-a441-6611214b30f7\") " pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd"
Jul 13 18:50:55.290748 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.290718    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a769d459-df42-47dc-8cfd-08a32d48bcea-serving-cert\") pod \"console-operator-7866497494-s92dx\" (UID: \"a769d459-df42-47dc-8cfd-08a32d48bcea\") " pod="openshift-console-operator/console-operator-7866497494-s92dx"
Jul 13 18:50:55.290911 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.290761    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-metrics-certs\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:50:55.290911 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.290783    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a769d459-df42-47dc-8cfd-08a32d48bcea-config\") pod \"console-operator-7866497494-s92dx\" (UID: \"a769d459-df42-47dc-8cfd-08a32d48bcea\") " pod="openshift-console-operator/console-operator-7866497494-s92dx"
Jul 13 18:50:55.290911 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.290805    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73a82b38-5c25-40f0-9849-4b0be568ae02-ca-trust-extracted\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.290911 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.290829    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-certificates\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.290911 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.290858    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhklf\" (UniqueName: \"kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-kube-api-access-lhklf\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.290911 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.290885    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr5kv\" (UniqueName: \"kubernetes.io/projected/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-kube-api-access-kr5kv\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:50:55.291387 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.290933    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-stats-auth\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:50:55.291387 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.290981    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73a82b38-5c25-40f0-9849-4b0be568ae02-trusted-ca\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.291387 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.291038    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-default-certificate\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:50:55.291387 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.291074    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a769d459-df42-47dc-8cfd-08a32d48bcea-trusted-ca\") pod \"console-operator-7866497494-s92dx\" (UID: \"a769d459-df42-47dc-8cfd-08a32d48bcea\") " pod="openshift-console-operator/console-operator-7866497494-s92dx"
Jul 13 18:50:55.291387 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.291101    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73a82b38-5c25-40f0-9849-4b0be568ae02-image-registry-private-configuration\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.291387 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.291134    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7zxw\" (UniqueName: \"kubernetes.io/projected/a769d459-df42-47dc-8cfd-08a32d48bcea-kube-api-access-z7zxw\") pod \"console-operator-7866497494-s92dx\" (UID: \"a769d459-df42-47dc-8cfd-08a32d48bcea\") " pod="openshift-console-operator/console-operator-7866497494-s92dx"
Jul 13 18:50:55.291387 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.291155    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73a82b38-5c25-40f0-9849-4b0be568ae02-installation-pull-secrets\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.291387 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.291208    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-service-ca-bundle\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:50:55.291803 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.291395    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-tls\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.291803 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.291442    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-bound-sa-token\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.291803 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.291489    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-certificates\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.291803 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:55.291535    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 18:50:55.291803 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:55.291550    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64f848497f-8c2qx: secret "image-registry-tls" not found
Jul 13 18:50:55.291803 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:55.291608    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-tls podName:73a82b38-5c25-40f0-9849-4b0be568ae02 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:55.791587706 +0000 UTC m=+77.305386850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-tls") pod "image-registry-64f848497f-8c2qx" (UID: "73a82b38-5c25-40f0-9849-4b0be568ae02") : secret "image-registry-tls" not found
Jul 13 18:50:55.292126 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.292014    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73a82b38-5c25-40f0-9849-4b0be568ae02-ca-trust-extracted\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.292494 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.292470    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73a82b38-5c25-40f0-9849-4b0be568ae02-trusted-ca\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.294673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.294651    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73a82b38-5c25-40f0-9849-4b0be568ae02-image-registry-private-configuration\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.294766 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.294747    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73a82b38-5c25-40f0-9849-4b0be568ae02-installation-pull-secrets\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.317070 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.317038    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-bound-sa-token\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.318383 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.318360    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhklf\" (UniqueName: \"kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-kube-api-access-lhklf\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.320091 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.320073    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd"
Jul 13 18:50:55.348591 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.348559    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-9pt94"]
Jul 13 18:50:55.351066 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:50:55.351039    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd176fb_f16d_4c0b_99ac_ef0bb25b1390.slice/crio-a7045bde8a3d2f25ee911d7bd83e60b18da11415c1519ec75e5a2749c9ea12d6 WatchSource:0}: Error finding container a7045bde8a3d2f25ee911d7bd83e60b18da11415c1519ec75e5a2749c9ea12d6: Status 404 returned error can't find the container with id a7045bde8a3d2f25ee911d7bd83e60b18da11415c1519ec75e5a2749c9ea12d6
Jul 13 18:50:55.403669 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.403634    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-9pt94" event={"ID":"7bd176fb-f16d-4c0b-99ac-ef0bb25b1390","Type":"ContainerStarted","Data":"a7045bde8a3d2f25ee911d7bd83e60b18da11415c1519ec75e5a2749c9ea12d6"}
Jul 13 18:50:55.458913 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.458844    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-tbqhd"]
Jul 13 18:50:55.461285 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:50:55.461260    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8977c40b_f5c4_4aea_a441_6611214b30f7.slice/crio-673da4f7cb7c9e93ec78a4f8901220df3739b1dc37841a995cf1551e284a1d93 WatchSource:0}: Error finding container 673da4f7cb7c9e93ec78a4f8901220df3739b1dc37841a995cf1551e284a1d93: Status 404 returned error can't find the container with id 673da4f7cb7c9e93ec78a4f8901220df3739b1dc37841a995cf1551e284a1d93
Jul 13 18:50:55.796879 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.796762    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-tls\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:55.797038 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:55.796916    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 18:50:55.797038 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:55.796934    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64f848497f-8c2qx: secret "image-registry-tls" not found
Jul 13 18:50:55.797038 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:55.796990    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-tls podName:73a82b38-5c25-40f0-9849-4b0be568ae02 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:56.79697442 +0000 UTC m=+78.310773567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-tls") pod "image-registry-64f848497f-8c2qx" (UID: "73a82b38-5c25-40f0-9849-4b0be568ae02") : secret "image-registry-tls" not found
Jul 13 18:50:55.929384 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.929353    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7jj6d\""
Jul 13 18:50:55.933923 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.933907    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 13 18:50:55.997608 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:55.996583    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 13 18:50:56.001916 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.001882    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a769d459-df42-47dc-8cfd-08a32d48bcea-trusted-ca\") pod \"console-operator-7866497494-s92dx\" (UID: \"a769d459-df42-47dc-8cfd-08a32d48bcea\") " pod="openshift-console-operator/console-operator-7866497494-s92dx"
Jul 13 18:50:56.020428 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.020397    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 13 18:50:56.021469 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:56.021451    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 13 18:50:56.021551 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:56.021531    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-metrics-certs podName:4fcfa1f0-d0ea-488a-a414-d09b8085aae5 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:56.521509385 +0000 UTC m=+78.035308528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-metrics-certs") pod "router-default-586b57644b-q24lm" (UID: "4fcfa1f0-d0ea-488a-a414-d09b8085aae5") : secret "router-metrics-certs-default" not found
Jul 13 18:50:56.037473 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.037444    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 13 18:50:56.044274 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.044252    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-default-certificate\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:50:56.058787 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.058721    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 13 18:50:56.068386 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.068362    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7zxw\" (UniqueName: \"kubernetes.io/projected/a769d459-df42-47dc-8cfd-08a32d48bcea-kube-api-access-z7zxw\") pod \"console-operator-7866497494-s92dx\" (UID: \"a769d459-df42-47dc-8cfd-08a32d48bcea\") " pod="openshift-console-operator/console-operator-7866497494-s92dx"
Jul 13 18:50:56.093013 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.092979    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 13 18:50:56.101855 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:56.101828    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-service-ca-bundle podName:4fcfa1f0-d0ea-488a-a414-d09b8085aae5 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:56.601807065 +0000 UTC m=+78.115606209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-service-ca-bundle") pod "router-default-586b57644b-q24lm" (UID: "4fcfa1f0-d0ea-488a-a414-d09b8085aae5") : configmap references non-existent config key: service-ca.crt
Jul 13 18:50:56.138232 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.138206    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-48thm\""
Jul 13 18:50:56.200493 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.200463    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 13 18:50:56.203692 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.203671    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a769d459-df42-47dc-8cfd-08a32d48bcea-serving-cert\") pod \"console-operator-7866497494-s92dx\" (UID: \"a769d459-df42-47dc-8cfd-08a32d48bcea\") " pod="openshift-console-operator/console-operator-7866497494-s92dx"
Jul 13 18:50:56.259407 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.259367    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 13 18:50:56.292074 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:56.292039    2569 secret.go:189] Couldn't get secret openshift-ingress/router-stats-default: failed to sync secret cache: timed out waiting for the condition
Jul 13 18:50:56.292260 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:56.292125    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-stats-auth podName:4fcfa1f0-d0ea-488a-a414-d09b8085aae5 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:56.792105489 +0000 UTC m=+78.305904637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "stats-auth" (UniqueName: "kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-stats-auth") pod "router-default-586b57644b-q24lm" (UID: "4fcfa1f0-d0ea-488a-a414-d09b8085aae5") : failed to sync secret cache: timed out waiting for the condition
Jul 13 18:50:56.292260 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:56.292039    2569 configmap.go:193] Couldn't get configMap openshift-console-operator/console-operator-config: failed to sync configmap cache: timed out waiting for the condition
Jul 13 18:50:56.292424 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:56.292266    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a769d459-df42-47dc-8cfd-08a32d48bcea-config podName:a769d459-df42-47dc-8cfd-08a32d48bcea nodeName:}" failed. No retries permitted until 2026-07-13 18:50:56.79224548 +0000 UTC m=+78.306044648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "config" (UniqueName: "kubernetes.io/configmap/a769d459-df42-47dc-8cfd-08a32d48bcea-config") pod "console-operator-7866497494-s92dx" (UID: "a769d459-df42-47dc-8cfd-08a32d48bcea") : failed to sync configmap cache: timed out waiting for the condition
Jul 13 18:50:56.407981 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.407937    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd" event={"ID":"8977c40b-f5c4-4aea-a441-6611214b30f7","Type":"ContainerStarted","Data":"673da4f7cb7c9e93ec78a4f8901220df3739b1dc37841a995cf1551e284a1d93"}
Jul 13 18:50:56.537152 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.537083    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 13 18:50:56.540071 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.540045    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr5kv\" (UniqueName: \"kubernetes.io/projected/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-kube-api-access-kr5kv\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:50:56.541054 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.541024    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 13 18:50:56.604490 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.604456    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-service-ca-bundle\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:50:56.604668 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.604520    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-metrics-certs\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:50:56.604668 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:56.604636    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-service-ca-bundle podName:4fcfa1f0-d0ea-488a-a414-d09b8085aae5 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:57.604621088 +0000 UTC m=+79.118420231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-service-ca-bundle") pod "router-default-586b57644b-q24lm" (UID: "4fcfa1f0-d0ea-488a-a414-d09b8085aae5") : configmap references non-existent config key: service-ca.crt
Jul 13 18:50:56.604769 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:56.604720    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 13 18:50:56.604808 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:56.604796    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-metrics-certs podName:4fcfa1f0-d0ea-488a-a414-d09b8085aae5 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:57.604778626 +0000 UTC m=+79.118577784 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-metrics-certs") pod "router-default-586b57644b-q24lm" (UID: "4fcfa1f0-d0ea-488a-a414-d09b8085aae5") : secret "router-metrics-certs-default" not found
Jul 13 18:50:56.661474 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.661385    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 13 18:50:56.806955 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.806886    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a769d459-df42-47dc-8cfd-08a32d48bcea-config\") pod \"console-operator-7866497494-s92dx\" (UID: \"a769d459-df42-47dc-8cfd-08a32d48bcea\") " pod="openshift-console-operator/console-operator-7866497494-s92dx"
Jul 13 18:50:56.806955 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.806946    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-stats-auth\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:50:56.807261 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.807141    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-tls\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:56.807398 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:56.807269    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 18:50:56.807398 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:56.807306    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64f848497f-8c2qx: secret "image-registry-tls" not found
Jul 13 18:50:56.807398 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:56.807383    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-tls podName:73a82b38-5c25-40f0-9849-4b0be568ae02 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:58.80736471 +0000 UTC m=+80.321163856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-tls") pod "image-registry-64f848497f-8c2qx" (UID: "73a82b38-5c25-40f0-9849-4b0be568ae02") : secret "image-registry-tls" not found
Jul 13 18:50:56.807638 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.807622    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a769d459-df42-47dc-8cfd-08a32d48bcea-config\") pod \"console-operator-7866497494-s92dx\" (UID: \"a769d459-df42-47dc-8cfd-08a32d48bcea\") " pod="openshift-console-operator/console-operator-7866497494-s92dx"
Jul 13 18:50:56.809962 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.809942    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-stats-auth\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:50:56.908601 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:56.908563    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-s92dx"
Jul 13 18:50:57.064589 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:57.064558    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-s92dx"]
Jul 13 18:50:57.067894 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:50:57.067862    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda769d459_df42_47dc_8cfd_08a32d48bcea.slice/crio-32352abd7a11b79eb1fd26b81e5c141ba2386580e755abbfc9ebf9c0019941f1 WatchSource:0}: Error finding container 32352abd7a11b79eb1fd26b81e5c141ba2386580e755abbfc9ebf9c0019941f1: Status 404 returned error can't find the container with id 32352abd7a11b79eb1fd26b81e5c141ba2386580e755abbfc9ebf9c0019941f1
Jul 13 18:50:57.412220 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:57.412180    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-s92dx" event={"ID":"a769d459-df42-47dc-8cfd-08a32d48bcea","Type":"ContainerStarted","Data":"32352abd7a11b79eb1fd26b81e5c141ba2386580e755abbfc9ebf9c0019941f1"}
Jul 13 18:50:57.613870 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:57.613601    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-metrics-certs\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:50:57.613870 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:57.613720    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-service-ca-bundle\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:50:57.613870 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:57.613756    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 13 18:50:57.613870 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:57.613853    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-metrics-certs podName:4fcfa1f0-d0ea-488a-a414-d09b8085aae5 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:59.613837261 +0000 UTC m=+81.127636405 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-metrics-certs") pod "router-default-586b57644b-q24lm" (UID: "4fcfa1f0-d0ea-488a-a414-d09b8085aae5") : secret "router-metrics-certs-default" not found
Jul 13 18:50:57.613870 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:57.613869    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-service-ca-bundle podName:4fcfa1f0-d0ea-488a-a414-d09b8085aae5 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:59.613862698 +0000 UTC m=+81.127661843 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-service-ca-bundle") pod "router-default-586b57644b-q24lm" (UID: "4fcfa1f0-d0ea-488a-a414-d09b8085aae5") : configmap references non-existent config key: service-ca.crt
Jul 13 18:50:58.823638 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:58.823554    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-tls\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:50:58.824053 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:58.823696    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 18:50:58.824053 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:58.823711    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64f848497f-8c2qx: secret "image-registry-tls" not found
Jul 13 18:50:58.824053 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:58.823757    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-tls podName:73a82b38-5c25-40f0-9849-4b0be568ae02 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:02.823745048 +0000 UTC m=+84.337544192 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-tls") pod "image-registry-64f848497f-8c2qx" (UID: "73a82b38-5c25-40f0-9849-4b0be568ae02") : secret "image-registry-tls" not found
Jul 13 18:50:59.429689 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.429643    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-9pt94" event={"ID":"7bd176fb-f16d-4c0b-99ac-ef0bb25b1390","Type":"ContainerStarted","Data":"060009844f402070d4269afcf28e4b77f51fe4c1befc3ab34cf65826655f106d"}
Jul 13 18:50:59.431544 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.431506    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd" event={"ID":"8977c40b-f5c4-4aea-a441-6611214b30f7","Type":"ContainerStarted","Data":"1f99bf2136fc57e74e231a5ba5a0423b67ec05c34283fcc048306503b5a15415"}
Jul 13 18:50:59.455491 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.455308    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-9pt94" podStartSLOduration=2.451447146 podStartE2EDuration="5.455271543s" totalImagesPullingTime="3.003824397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:59.454111423 +0000 UTC m=+80.967910590" watchObservedRunningTime="2026-07-13 18:50:59.455271543 +0000 UTC m=+80.969070710"
Jul 13 18:50:59.477839 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.477776    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd" podStartSLOduration=2.582588378 podStartE2EDuration="5.477756071s" totalImagesPullingTime="2.895167693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:59.47650542 +0000 UTC m=+80.990304586" watchObservedRunningTime="2026-07-13 18:50:59.477756071 +0000 UTC m=+80.991555239"
Jul 13 18:50:59.632068 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.632016    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-service-ca-bundle\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:50:59.632239 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.632110    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-metrics-certs\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:50:59.632239 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:59.632203    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-service-ca-bundle podName:4fcfa1f0-d0ea-488a-a414-d09b8085aae5 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:03.632180039 +0000 UTC m=+85.145979193 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-service-ca-bundle") pod "router-default-586b57644b-q24lm" (UID: "4fcfa1f0-d0ea-488a-a414-d09b8085aae5") : configmap references non-existent config key: service-ca.crt
Jul 13 18:50:59.632359 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:59.632256    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 13 18:50:59.632359 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:59.632334    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-metrics-certs podName:4fcfa1f0-d0ea-488a-a414-d09b8085aae5 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:03.63231903 +0000 UTC m=+85.146118174 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-metrics-certs") pod "router-default-586b57644b-q24lm" (UID: "4fcfa1f0-d0ea-488a-a414-d09b8085aae5") : secret "router-metrics-certs-default" not found
Jul 13 18:50:59.753836 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.753755    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2h9jt"]
Jul 13 18:50:59.758863 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.758830    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:50:59.764516 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.764491    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 13 18:50:59.764711 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.764698    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c5j2r\""
Jul 13 18:50:59.764866 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.764854    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 13 18:50:59.777914 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.777886    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2h9jt"]
Jul 13 18:50:59.833813 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.833791    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/929eec72-b408-4af0-9f6f-8e140d648876-crio-socket\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:50:59.834078 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.833845    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/929eec72-b408-4af0-9f6f-8e140d648876-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:50:59.834078 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.833868    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d98m7\" (UniqueName: \"kubernetes.io/projected/929eec72-b408-4af0-9f6f-8e140d648876-kube-api-access-d98m7\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:50:59.834078 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.833928    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/929eec72-b408-4af0-9f6f-8e140d648876-data-volume\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:50:59.834078 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.833981    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/929eec72-b408-4af0-9f6f-8e140d648876-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:50:59.934452 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.934410    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d98m7\" (UniqueName: \"kubernetes.io/projected/929eec72-b408-4af0-9f6f-8e140d648876-kube-api-access-d98m7\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:50:59.934648 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.934497    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/929eec72-b408-4af0-9f6f-8e140d648876-data-volume\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:50:59.934648 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.934535    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/929eec72-b408-4af0-9f6f-8e140d648876-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:50:59.934648 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.934606    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/929eec72-b408-4af0-9f6f-8e140d648876-crio-socket\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:50:59.934777 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.934673    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/929eec72-b408-4af0-9f6f-8e140d648876-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:50:59.934777 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:59.934703    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 18:50:59.934777 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.934754    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/929eec72-b408-4af0-9f6f-8e140d648876-crio-socket\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:50:59.934894 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:59.934768    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/929eec72-b408-4af0-9f6f-8e140d648876-kube-rbac-proxy-cm podName:929eec72-b408-4af0-9f6f-8e140d648876 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:00.434749302 +0000 UTC m=+81.948548446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/929eec72-b408-4af0-9f6f-8e140d648876-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2h9jt" (UID: "929eec72-b408-4af0-9f6f-8e140d648876") : configmap "kube-rbac-proxy" not found
Jul 13 18:50:59.934894 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:59.934779    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 18:50:59.934894 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:50:59.934865    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/929eec72-b408-4af0-9f6f-8e140d648876-insights-runtime-extractor-tls podName:929eec72-b408-4af0-9f6f-8e140d648876 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:00.434846171 +0000 UTC m=+81.948645326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/929eec72-b408-4af0-9f6f-8e140d648876-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2h9jt" (UID: "929eec72-b408-4af0-9f6f-8e140d648876") : secret "insights-runtime-extractor-tls" not found
Jul 13 18:50:59.934894 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.934876    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/929eec72-b408-4af0-9f6f-8e140d648876-data-volume\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:50:59.945958 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:50:59.945917    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d98m7\" (UniqueName: \"kubernetes.io/projected/929eec72-b408-4af0-9f6f-8e140d648876-kube-api-access-d98m7\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:51:00.436056 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:00.436024    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/0.log"
Jul 13 18:51:00.436232 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:00.436072    2569 generic.go:356] "Generic (PLEG): container finished" podID="a769d459-df42-47dc-8cfd-08a32d48bcea" containerID="6b90f2f6f4be9b236be0ce9d5d6b2bd907920412d3251f613b89d16ccc1cd075" exitCode=255
Jul 13 18:51:00.436232 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:00.436162    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-s92dx" event={"ID":"a769d459-df42-47dc-8cfd-08a32d48bcea","Type":"ContainerDied","Data":"6b90f2f6f4be9b236be0ce9d5d6b2bd907920412d3251f613b89d16ccc1cd075"}
Jul 13 18:51:00.436390 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:00.436374    2569 scope.go:117] "RemoveContainer" containerID="6b90f2f6f4be9b236be0ce9d5d6b2bd907920412d3251f613b89d16ccc1cd075"
Jul 13 18:51:00.438457 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:00.438439    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/929eec72-b408-4af0-9f6f-8e140d648876-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:51:00.438548 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:00.438513    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/929eec72-b408-4af0-9f6f-8e140d648876-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:51:00.438589 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:00.438545    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 18:51:00.438631 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:00.438602    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/929eec72-b408-4af0-9f6f-8e140d648876-kube-rbac-proxy-cm podName:929eec72-b408-4af0-9f6f-8e140d648876 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:01.438587808 +0000 UTC m=+82.952386952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/929eec72-b408-4af0-9f6f-8e140d648876-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2h9jt" (UID: "929eec72-b408-4af0-9f6f-8e140d648876") : configmap "kube-rbac-proxy" not found
Jul 13 18:51:00.438631 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:00.438603    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:00.438722 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:00.438650    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/929eec72-b408-4af0-9f6f-8e140d648876-insights-runtime-extractor-tls podName:929eec72-b408-4af0-9f6f-8e140d648876 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:01.438635748 +0000 UTC m=+82.952434912 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/929eec72-b408-4af0-9f6f-8e140d648876-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2h9jt" (UID: "929eec72-b408-4af0-9f6f-8e140d648876") : secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:01.441620 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:01.441588    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 18:51:01.442157 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:01.441988    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/0.log"
Jul 13 18:51:01.442157 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:01.442033    2569 generic.go:356] "Generic (PLEG): container finished" podID="a769d459-df42-47dc-8cfd-08a32d48bcea" containerID="619e1a8a205566cb8a66fa37c174f5c7c40011f8bb30c4ddcc601ab0407037a7" exitCode=255
Jul 13 18:51:01.442157 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:01.442116    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-s92dx" event={"ID":"a769d459-df42-47dc-8cfd-08a32d48bcea","Type":"ContainerDied","Data":"619e1a8a205566cb8a66fa37c174f5c7c40011f8bb30c4ddcc601ab0407037a7"}
Jul 13 18:51:01.442351 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:01.442170    2569 scope.go:117] "RemoveContainer" containerID="6b90f2f6f4be9b236be0ce9d5d6b2bd907920412d3251f613b89d16ccc1cd075"
Jul 13 18:51:01.442413 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:01.442371    2569 scope.go:117] "RemoveContainer" containerID="619e1a8a205566cb8a66fa37c174f5c7c40011f8bb30c4ddcc601ab0407037a7"
Jul 13 18:51:01.442574 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:01.442545    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-s92dx_openshift-console-operator(a769d459-df42-47dc-8cfd-08a32d48bcea)\"" pod="openshift-console-operator/console-operator-7866497494-s92dx" podUID="a769d459-df42-47dc-8cfd-08a32d48bcea"
Jul 13 18:51:01.447613 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:01.447590    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/929eec72-b408-4af0-9f6f-8e140d648876-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:51:01.447684 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:01.447646    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/929eec72-b408-4af0-9f6f-8e140d648876-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:51:01.447747 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:01.447731    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:01.447747 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:01.447741    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 18:51:01.447817 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:01.447794    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/929eec72-b408-4af0-9f6f-8e140d648876-insights-runtime-extractor-tls podName:929eec72-b408-4af0-9f6f-8e140d648876 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:03.447774856 +0000 UTC m=+84.961574026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/929eec72-b408-4af0-9f6f-8e140d648876-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2h9jt" (UID: "929eec72-b408-4af0-9f6f-8e140d648876") : secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:01.447817 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:01.447809    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/929eec72-b408-4af0-9f6f-8e140d648876-kube-rbac-proxy-cm podName:929eec72-b408-4af0-9f6f-8e140d648876 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:03.447802792 +0000 UTC m=+84.961601936 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/929eec72-b408-4af0-9f6f-8e140d648876-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2h9jt" (UID: "929eec72-b408-4af0-9f6f-8e140d648876") : configmap "kube-rbac-proxy" not found
Jul 13 18:51:01.668573 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:01.668544    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vxf4w_3963e434-e4ab-4d73-b496-2472c7711891/dns-node-resolver/0.log"
Jul 13 18:51:02.446887 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:02.446855    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 18:51:02.447273 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:02.447189    2569 scope.go:117] "RemoveContainer" containerID="619e1a8a205566cb8a66fa37c174f5c7c40011f8bb30c4ddcc601ab0407037a7"
Jul 13 18:51:02.447366 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:02.447345    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-s92dx_openshift-console-operator(a769d459-df42-47dc-8cfd-08a32d48bcea)\"" pod="openshift-console-operator/console-operator-7866497494-s92dx" podUID="a769d459-df42-47dc-8cfd-08a32d48bcea"
Jul 13 18:51:02.472173 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:02.472144    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gdf7f_ddda9308-3395-4d7b-af21-a398125bd512/node-ca/0.log"
Jul 13 18:51:02.860588 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:02.860556    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-tls\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:51:02.860763 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:02.860712    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 18:51:02.860763 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:02.860733    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64f848497f-8c2qx: secret "image-registry-tls" not found
Jul 13 18:51:02.860841 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:02.860799    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-tls podName:73a82b38-5c25-40f0-9849-4b0be568ae02 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:10.860783363 +0000 UTC m=+92.374582511 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-tls") pod "image-registry-64f848497f-8c2qx" (UID: "73a82b38-5c25-40f0-9849-4b0be568ae02") : secret "image-registry-tls" not found
Jul 13 18:51:03.465005 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:03.464976    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/929eec72-b408-4af0-9f6f-8e140d648876-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:51:03.465410 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:03.465033    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/929eec72-b408-4af0-9f6f-8e140d648876-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:51:03.465410 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:03.465101    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 18:51:03.465410 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:03.465132    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:03.465410 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:03.465153    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/929eec72-b408-4af0-9f6f-8e140d648876-kube-rbac-proxy-cm podName:929eec72-b408-4af0-9f6f-8e140d648876 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:07.465140531 +0000 UTC m=+88.978939674 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/929eec72-b408-4af0-9f6f-8e140d648876-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2h9jt" (UID: "929eec72-b408-4af0-9f6f-8e140d648876") : configmap "kube-rbac-proxy" not found
Jul 13 18:51:03.465410 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:03.465184    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/929eec72-b408-4af0-9f6f-8e140d648876-insights-runtime-extractor-tls podName:929eec72-b408-4af0-9f6f-8e140d648876 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:07.465170995 +0000 UTC m=+88.978970138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/929eec72-b408-4af0-9f6f-8e140d648876-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2h9jt" (UID: "929eec72-b408-4af0-9f6f-8e140d648876") : secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:03.469606 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:03.469588    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9qzhl_502189f5-f517-4aa2-b359-07b8c5be3b9a/serve-healthcheck-canary/0.log"
Jul 13 18:51:03.667218 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:03.667185    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-service-ca-bundle\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:51:03.667414 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:03.667244    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-metrics-certs\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:51:03.667414 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:03.667375    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-service-ca-bundle podName:4fcfa1f0-d0ea-488a-a414-d09b8085aae5 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:11.667355462 +0000 UTC m=+93.181154606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-service-ca-bundle") pod "router-default-586b57644b-q24lm" (UID: "4fcfa1f0-d0ea-488a-a414-d09b8085aae5") : configmap references non-existent config key: service-ca.crt
Jul 13 18:51:03.667414 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:03.667406    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 13 18:51:03.667549 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:03.667462    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-metrics-certs podName:4fcfa1f0-d0ea-488a-a414-d09b8085aae5 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:11.667450661 +0000 UTC m=+93.181249809 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-metrics-certs") pod "router-default-586b57644b-q24lm" (UID: "4fcfa1f0-d0ea-488a-a414-d09b8085aae5") : secret "router-metrics-certs-default" not found
Jul 13 18:51:06.908722 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:06.908678    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-s92dx"
Jul 13 18:51:06.909231 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:06.908749    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-s92dx"
Jul 13 18:51:06.909231 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:06.909169    2569 scope.go:117] "RemoveContainer" containerID="619e1a8a205566cb8a66fa37c174f5c7c40011f8bb30c4ddcc601ab0407037a7"
Jul 13 18:51:06.909422 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:06.909398    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-s92dx_openshift-console-operator(a769d459-df42-47dc-8cfd-08a32d48bcea)\"" pod="openshift-console-operator/console-operator-7866497494-s92dx" podUID="a769d459-df42-47dc-8cfd-08a32d48bcea"
Jul 13 18:51:07.500589 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:07.500550    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/929eec72-b408-4af0-9f6f-8e140d648876-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:51:07.500823 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:07.500647    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/929eec72-b408-4af0-9f6f-8e140d648876-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:51:07.500823 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:07.500723    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:07.500823 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:07.500746    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 18:51:07.500823 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:07.500803    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/929eec72-b408-4af0-9f6f-8e140d648876-insights-runtime-extractor-tls podName:929eec72-b408-4af0-9f6f-8e140d648876 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:15.500784702 +0000 UTC m=+97.014583865 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/929eec72-b408-4af0-9f6f-8e140d648876-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2h9jt" (UID: "929eec72-b408-4af0-9f6f-8e140d648876") : secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:07.500823 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:07.500819    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/929eec72-b408-4af0-9f6f-8e140d648876-kube-rbac-proxy-cm podName:929eec72-b408-4af0-9f6f-8e140d648876 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:15.500812669 +0000 UTC m=+97.014611813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/929eec72-b408-4af0-9f6f-8e140d648876-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2h9jt" (UID: "929eec72-b408-4af0-9f6f-8e140d648876") : configmap "kube-rbac-proxy" not found
Jul 13 18:51:10.931466 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:10.931427    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-tls\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:51:10.934138 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:10.934104    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-tls\") pod \"image-registry-64f848497f-8c2qx\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") " pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:51:11.036673 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:11.036634    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:51:11.180410 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:11.180377    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64f848497f-8c2qx"]
Jul 13 18:51:11.183210 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:51:11.183138    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a82b38_5c25_40f0_9849_4b0be568ae02.slice/crio-3c4c8bf6f74e7052a10044b1d2e55ce0646cb61c140c51d1ba524cd67416528c WatchSource:0}: Error finding container 3c4c8bf6f74e7052a10044b1d2e55ce0646cb61c140c51d1ba524cd67416528c: Status 404 returned error can't find the container with id 3c4c8bf6f74e7052a10044b1d2e55ce0646cb61c140c51d1ba524cd67416528c
Jul 13 18:51:11.476855 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:11.476758    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64f848497f-8c2qx" event={"ID":"73a82b38-5c25-40f0-9849-4b0be568ae02","Type":"ContainerStarted","Data":"b32f9f2b0b9d5075f220f82e05fd6b4cf75138fc9326bc3b1760e2010cf09712"}
Jul 13 18:51:11.476855 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:11.476823    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64f848497f-8c2qx" event={"ID":"73a82b38-5c25-40f0-9849-4b0be568ae02","Type":"ContainerStarted","Data":"3c4c8bf6f74e7052a10044b1d2e55ce0646cb61c140c51d1ba524cd67416528c"}
Jul 13 18:51:11.477078 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:11.476864    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:51:11.502891 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:11.502832    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64f848497f-8c2qx" podStartSLOduration=16.502816701 podStartE2EDuration="16.502816701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:11.500741685 +0000 UTC m=+93.014540852" watchObservedRunningTime="2026-07-13 18:51:11.502816701 +0000 UTC m=+93.016615904"
Jul 13 18:51:11.738813 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:11.738709    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-service-ca-bundle\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:51:11.738813 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:11.738774    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-metrics-certs\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:51:11.739407 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:11.739382    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-service-ca-bundle\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:51:11.741170 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:11.741143    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fcfa1f0-d0ea-488a-a414-d09b8085aae5-metrics-certs\") pod \"router-default-586b57644b-q24lm\" (UID: \"4fcfa1f0-d0ea-488a-a414-d09b8085aae5\") " pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:51:11.917121 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:11.917072    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:51:12.055952 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:12.055928    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-586b57644b-q24lm"]
Jul 13 18:51:12.057654 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:51:12.057626    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fcfa1f0_d0ea_488a_a414_d09b8085aae5.slice/crio-52d1f40f53a4fa02a2d9f3063b6720c399b46d1fa6bef7d26bf4d139b8908b76 WatchSource:0}: Error finding container 52d1f40f53a4fa02a2d9f3063b6720c399b46d1fa6bef7d26bf4d139b8908b76: Status 404 returned error can't find the container with id 52d1f40f53a4fa02a2d9f3063b6720c399b46d1fa6bef7d26bf4d139b8908b76
Jul 13 18:51:12.481726 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:12.481684    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-586b57644b-q24lm" event={"ID":"4fcfa1f0-d0ea-488a-a414-d09b8085aae5","Type":"ContainerStarted","Data":"d8ef3ffdde6949ed16d5a95e996de18d608110488eecf8d9965b52bca9c15152"}
Jul 13 18:51:12.481929 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:12.481752    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-586b57644b-q24lm" event={"ID":"4fcfa1f0-d0ea-488a-a414-d09b8085aae5","Type":"ContainerStarted","Data":"52d1f40f53a4fa02a2d9f3063b6720c399b46d1fa6bef7d26bf4d139b8908b76"}
Jul 13 18:51:12.508980 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:12.508933    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-586b57644b-q24lm" podStartSLOduration=17.508917417 podStartE2EDuration="17.508917417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:12.507470884 +0000 UTC m=+94.021270051" watchObservedRunningTime="2026-07-13 18:51:12.508917417 +0000 UTC m=+94.022716582"
Jul 13 18:51:12.917861 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:12.917826    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:51:12.920272 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:12.920250    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:51:13.484447 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:13.484417    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:51:13.485638 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:13.485610    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-586b57644b-q24lm"
Jul 13 18:51:15.573912 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:15.573858    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/929eec72-b408-4af0-9f6f-8e140d648876-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:51:15.574433 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:15.574004    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/929eec72-b408-4af0-9f6f-8e140d648876-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:51:15.574491 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:15.574457    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/929eec72-b408-4af0-9f6f-8e140d648876-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:51:15.576403 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:15.576380    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/929eec72-b408-4af0-9f6f-8e140d648876-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2h9jt\" (UID: \"929eec72-b408-4af0-9f6f-8e140d648876\") " pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:51:15.669877 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:15.669839    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2h9jt"
Jul 13 18:51:15.805510 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:15.805328    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2h9jt"]
Jul 13 18:51:15.807906 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:51:15.807873    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929eec72_b408_4af0_9f6f_8e140d648876.slice/crio-6d37df6cd6994d4fd87135949adb63c061c9f9f3f3534ba05608d00eb532494e WatchSource:0}: Error finding container 6d37df6cd6994d4fd87135949adb63c061c9f9f3f3534ba05608d00eb532494e: Status 404 returned error can't find the container with id 6d37df6cd6994d4fd87135949adb63c061c9f9f3f3534ba05608d00eb532494e
Jul 13 18:51:16.497677 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:16.497634    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2h9jt" event={"ID":"929eec72-b408-4af0-9f6f-8e140d648876","Type":"ContainerStarted","Data":"6d37df6cd6994d4fd87135949adb63c061c9f9f3f3534ba05608d00eb532494e"}
Jul 13 18:51:18.055312 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:18.055264    2569 scope.go:117] "RemoveContainer" containerID="619e1a8a205566cb8a66fa37c174f5c7c40011f8bb30c4ddcc601ab0407037a7"
Jul 13 18:51:19.514261 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:19.514114    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 18:51:19.514261 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:19.514224    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-s92dx" event={"ID":"a769d459-df42-47dc-8cfd-08a32d48bcea","Type":"ContainerStarted","Data":"35c112b8c80c7fc95cc803ac9b3e75fdd6643b4c7709c37cc07449818020b191"}
Jul 13 18:51:19.514747 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:19.514579    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-s92dx"
Jul 13 18:51:19.516074 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:19.516044    2569 patch_prober.go:28] interesting pod/console-operator-7866497494-s92dx container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.10:8443/readyz\": dial tcp 10.133.0.10:8443: connect: connection refused" start-of-body=
Jul 13 18:51:19.516185 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:19.516100    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7866497494-s92dx" podUID="a769d459-df42-47dc-8cfd-08a32d48bcea" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.10:8443/readyz\": dial tcp 10.133.0.10:8443: connect: connection refused"
Jul 13 18:51:19.539255 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:19.539211    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-s92dx" podStartSLOduration=21.775405418 podStartE2EDuration="24.539193893s" totalImagesPullingTime="2.763788475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:19.539054164 +0000 UTC m=+101.052853326" watchObservedRunningTime="2026-07-13 18:51:19.539193893 +0000 UTC m=+101.052993058"
Jul 13 18:51:19.811016 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:19.810919    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls\") pod \"dns-default-xzzrg\" (UID: \"c4789bc2-8df6-4746-b737-e6f4fd4a9f99\") " pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:51:19.813396 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:19.813359    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4789bc2-8df6-4746-b737-e6f4fd4a9f99-metrics-tls\") pod \"dns-default-xzzrg\" (UID: \"c4789bc2-8df6-4746-b737-e6f4fd4a9f99\") " pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:51:19.824362 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:19.824273    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-88cff\""
Jul 13 18:51:19.832299 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:19.832267    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:51:19.987216 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:19.987186    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xzzrg"]
Jul 13 18:51:19.990188 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:51:19.990162    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4789bc2_8df6_4746_b737_e6f4fd4a9f99.slice/crio-e9b5ec2d03d4babf28ad8aba7d85b277aeb5ed93208d59c92b51718bfd553a10 WatchSource:0}: Error finding container e9b5ec2d03d4babf28ad8aba7d85b277aeb5ed93208d59c92b51718bfd553a10: Status 404 returned error can't find the container with id e9b5ec2d03d4babf28ad8aba7d85b277aeb5ed93208d59c92b51718bfd553a10
Jul 13 18:51:20.520349 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:20.520315    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2h9jt" event={"ID":"929eec72-b408-4af0-9f6f-8e140d648876","Type":"ContainerStarted","Data":"2e6b54388252849380a51902c37ba0ca34ec0ff7a8fe81847abb785a5529a40a"}
Jul 13 18:51:20.520760 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:20.520386    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2h9jt" event={"ID":"929eec72-b408-4af0-9f6f-8e140d648876","Type":"ContainerStarted","Data":"3f79e8519514f91b1f3fe2e22c816e924f9872a62a538b72f6233824a3fd317a"}
Jul 13 18:51:20.521670 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:20.521643    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xzzrg" event={"ID":"c4789bc2-8df6-4746-b737-e6f4fd4a9f99","Type":"ContainerStarted","Data":"e9b5ec2d03d4babf28ad8aba7d85b277aeb5ed93208d59c92b51718bfd553a10"}
Jul 13 18:51:20.526059 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:20.526037    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-s92dx"
Jul 13 18:51:21.394727 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:21.394680    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xrtxq"
Jul 13 18:51:22.532128 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:22.532088    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xzzrg" event={"ID":"c4789bc2-8df6-4746-b737-e6f4fd4a9f99","Type":"ContainerStarted","Data":"e278d5dc4ede4fd027f662199216f7146a69ef108dd7b27b3f5adf10ae3d5ded"}
Jul 13 18:51:22.532589 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:22.532145    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xzzrg" event={"ID":"c4789bc2-8df6-4746-b737-e6f4fd4a9f99","Type":"ContainerStarted","Data":"af53d26598768cd60f19f4b0a77fc468c03671ab351ff02c66f5b3e3b824c4e8"}
Jul 13 18:51:22.532589 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:22.532249    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:51:22.554527 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:22.554468    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xzzrg" podStartSLOduration=65.916334629 podStartE2EDuration="1m7.554444835s" totalImagesPullingTime="1.638110206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:22.553598028 +0000 UTC m=+104.067397195" watchObservedRunningTime="2026-07-13 18:51:22.554444835 +0000 UTC m=+104.068244003"
Jul 13 18:51:24.540950 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:24.540916    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2h9jt" event={"ID":"929eec72-b408-4af0-9f6f-8e140d648876","Type":"ContainerStarted","Data":"1772f3c7c06bab71161887c4b3e92f5f11e6c4c8df568df6c239062fb25e076b"}
Jul 13 18:51:24.563711 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:24.563659    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2h9jt" podStartSLOduration=17.939761365 podStartE2EDuration="25.563644365s" totalImagesPullingTime="7.623883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:24.562347815 +0000 UTC m=+106.076146981" watchObservedRunningTime="2026-07-13 18:51:24.563644365 +0000 UTC m=+106.077443530"
Jul 13 18:51:27.966389 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:27.966357    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-9tnzg"]
Jul 13 18:51:27.969277 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:27.969257    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-9tnzg"
Jul 13 18:51:27.975321 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:27.975074    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxh8n\" (UniqueName: \"kubernetes.io/projected/92c2bd01-a76e-476c-b074-3a33fd725f90-kube-api-access-vxh8n\") pod \"downloads-68c9cbb497-9tnzg\" (UID: \"92c2bd01-a76e-476c-b074-3a33fd725f90\") " pod="openshift-console/downloads-68c9cbb497-9tnzg"
Jul 13 18:51:27.979274 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:27.979257    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 13 18:51:27.979814 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:27.979797    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-mtg2g\""
Jul 13 18:51:27.980094 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:27.980076    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 13 18:51:27.987792 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:27.987773    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-9tnzg"]
Jul 13 18:51:28.010720 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:28.010688    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64f848497f-8c2qx"]
Jul 13 18:51:28.015379 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:28.015352    2569 patch_prober.go:28] interesting pod/image-registry-64f848497f-8c2qx container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 13 18:51:28.015517 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:28.015401    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-64f848497f-8c2qx" podUID="73a82b38-5c25-40f0-9849-4b0be568ae02" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 18:51:28.076017 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:28.075985    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxh8n\" (UniqueName: \"kubernetes.io/projected/92c2bd01-a76e-476c-b074-3a33fd725f90-kube-api-access-vxh8n\") pod \"downloads-68c9cbb497-9tnzg\" (UID: \"92c2bd01-a76e-476c-b074-3a33fd725f90\") " pod="openshift-console/downloads-68c9cbb497-9tnzg"
Jul 13 18:51:28.089613 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:28.089587    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxh8n\" (UniqueName: \"kubernetes.io/projected/92c2bd01-a76e-476c-b074-3a33fd725f90-kube-api-access-vxh8n\") pod \"downloads-68c9cbb497-9tnzg\" (UID: \"92c2bd01-a76e-476c-b074-3a33fd725f90\") " pod="openshift-console/downloads-68c9cbb497-9tnzg"
Jul 13 18:51:28.277236 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:28.277141    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-9tnzg"
Jul 13 18:51:28.417627 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:51:28.417596    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c2bd01_a76e_476c_b074_3a33fd725f90.slice/crio-965b254aeb9c4be6b8fb5f6607f2b470862d71ea306a2de8114349a8b2937c78 WatchSource:0}: Error finding container 965b254aeb9c4be6b8fb5f6607f2b470862d71ea306a2de8114349a8b2937c78: Status 404 returned error can't find the container with id 965b254aeb9c4be6b8fb5f6607f2b470862d71ea306a2de8114349a8b2937c78
Jul 13 18:51:28.420346 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:28.420321    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-9tnzg"]
Jul 13 18:51:28.554464 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:28.554383    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-9tnzg" event={"ID":"92c2bd01-a76e-476c-b074-3a33fd725f90","Type":"ContainerStarted","Data":"965b254aeb9c4be6b8fb5f6607f2b470862d71ea306a2de8114349a8b2937c78"}
Jul 13 18:51:30.184283 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.184245    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7845d98b48-vhgkl"]
Jul 13 18:51:30.188421 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.188389    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:30.197978 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.197954    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 13 18:51:30.198114 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.197991    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 13 18:51:30.202220 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.202199    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-gxdrl\""
Jul 13 18:51:30.202854 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.202830    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 13 18:51:30.202954 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.202832    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 13 18:51:30.207693 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.207671    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 13 18:51:30.219081 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.219058    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7845d98b48-vhgkl"]
Jul 13 18:51:30.293763 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.293728    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-oauth-serving-cert\") pod \"console-7845d98b48-vhgkl\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") " pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:30.293940 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.293777    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-service-ca\") pod \"console-7845d98b48-vhgkl\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") " pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:30.293940 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.293827    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-console-serving-cert\") pod \"console-7845d98b48-vhgkl\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") " pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:30.293940 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.293851    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-console-oauth-config\") pod \"console-7845d98b48-vhgkl\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") " pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:30.293940 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.293912    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-console-config\") pod \"console-7845d98b48-vhgkl\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") " pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:30.293940 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.293930    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv6bx\" (UniqueName: \"kubernetes.io/projected/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-kube-api-access-sv6bx\") pod \"console-7845d98b48-vhgkl\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") " pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:30.394696 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.394660    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-console-serving-cert\") pod \"console-7845d98b48-vhgkl\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") " pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:30.394696 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.394700    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-console-oauth-config\") pod \"console-7845d98b48-vhgkl\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") " pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:30.394950 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.394760    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-console-config\") pod \"console-7845d98b48-vhgkl\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") " pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:30.394950 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.394788    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv6bx\" (UniqueName: \"kubernetes.io/projected/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-kube-api-access-sv6bx\") pod \"console-7845d98b48-vhgkl\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") " pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:30.394950 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.394823    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-oauth-serving-cert\") pod \"console-7845d98b48-vhgkl\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") " pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:30.394950 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.394923    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-service-ca\") pod \"console-7845d98b48-vhgkl\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") " pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:30.395649 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.395622    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-console-config\") pod \"console-7845d98b48-vhgkl\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") " pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:30.395922 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.395630    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-oauth-serving-cert\") pod \"console-7845d98b48-vhgkl\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") " pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:30.395922 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.395668    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-service-ca\") pod \"console-7845d98b48-vhgkl\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") " pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:30.398251 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.398230    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-console-oauth-config\") pod \"console-7845d98b48-vhgkl\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") " pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:30.398374 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.398306    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-console-serving-cert\") pod \"console-7845d98b48-vhgkl\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") " pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:30.406005 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.405981    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv6bx\" (UniqueName: \"kubernetes.io/projected/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-kube-api-access-sv6bx\") pod \"console-7845d98b48-vhgkl\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") " pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:30.419840 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.419808    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-r78ll"]
Jul 13 18:51:30.423283 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.423261    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-r78ll"
Jul 13 18:51:30.426333 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:30.426306    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"prometheus-operator-admission-webhook-dockercfg-bvlx4\" is forbidden: User \"system:node:ip-10-0-139-134.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-139-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bvlx4\"" type="*v1.Secret"
Jul 13 18:51:30.426470 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:30.426440    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"prometheus-operator-admission-webhook-77d98bc786-r78ll\" is forbidden: User \"system:node:ip-10-0-139-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-139-134.ec2.internal' and this object" podUID="25d90578-2923-49f9-88e8-cf9f72a0ebb1" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-r78ll"
Jul 13 18:51:30.426795 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:30.426758    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"prometheus-operator-admission-webhook-tls\" is forbidden: User \"system:node:ip-10-0-139-134.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-139-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" type="*v1.Secret"
Jul 13 18:51:30.441462 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.441365    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-r78ll"]
Jul 13 18:51:30.496071 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.496029    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/25d90578-2923-49f9-88e8-cf9f72a0ebb1-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-r78ll\" (UID: \"25d90578-2923-49f9-88e8-cf9f72a0ebb1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-r78ll"
Jul 13 18:51:30.498978 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.498944    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:30.596990 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.596951    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/25d90578-2923-49f9-88e8-cf9f72a0ebb1-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-r78ll\" (UID: \"25d90578-2923-49f9-88e8-cf9f72a0ebb1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-r78ll"
Jul 13 18:51:30.650683 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:30.650653    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7845d98b48-vhgkl"]
Jul 13 18:51:30.654031 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:51:30.654004    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668d7b34_95bb_46ab_a0ba_1d5dd4911ddf.slice/crio-b1d18b48a22a6daf799a2c4d445557d2289e2a187f3117805c2ff5084e32242f WatchSource:0}: Error finding container b1d18b48a22a6daf799a2c4d445557d2289e2a187f3117805c2ff5084e32242f: Status 404 returned error can't find the container with id b1d18b48a22a6daf799a2c4d445557d2289e2a187f3117805c2ff5084e32242f
Jul 13 18:51:31.486541 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:31.486505    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bvlx4\""
Jul 13 18:51:31.569819 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:31.569779    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7845d98b48-vhgkl" event={"ID":"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf","Type":"ContainerStarted","Data":"b1d18b48a22a6daf799a2c4d445557d2289e2a187f3117805c2ff5084e32242f"}
Jul 13 18:51:31.597544 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:31.597509    2569 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: failed to sync secret cache: timed out waiting for the condition
Jul 13 18:51:31.597725 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:31.597615    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25d90578-2923-49f9-88e8-cf9f72a0ebb1-tls-certificates podName:25d90578-2923-49f9-88e8-cf9f72a0ebb1 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:32.097582532 +0000 UTC m=+113.611381675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/25d90578-2923-49f9-88e8-cf9f72a0ebb1-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-r78ll" (UID: "25d90578-2923-49f9-88e8-cf9f72a0ebb1") : failed to sync secret cache: timed out waiting for the condition
Jul 13 18:51:31.726097 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:31.726059    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 13 18:51:32.110497 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:32.110462    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/25d90578-2923-49f9-88e8-cf9f72a0ebb1-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-r78ll\" (UID: \"25d90578-2923-49f9-88e8-cf9f72a0ebb1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-r78ll"
Jul 13 18:51:32.114679 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:32.114652    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/25d90578-2923-49f9-88e8-cf9f72a0ebb1-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-r78ll\" (UID: \"25d90578-2923-49f9-88e8-cf9f72a0ebb1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-r78ll"
Jul 13 18:51:32.234511 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:32.234468    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-r78ll"
Jul 13 18:51:32.412670 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:32.411277    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-r78ll"]
Jul 13 18:51:32.416317 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:51:32.416245    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d90578_2923_49f9_88e8_cf9f72a0ebb1.slice/crio-59ba94b24b6f89fbad65ac6a58f864412f92678e497416cba2020065b58ef961 WatchSource:0}: Error finding container 59ba94b24b6f89fbad65ac6a58f864412f92678e497416cba2020065b58ef961: Status 404 returned error can't find the container with id 59ba94b24b6f89fbad65ac6a58f864412f92678e497416cba2020065b58ef961
Jul 13 18:51:32.538908 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:32.538671    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xzzrg"
Jul 13 18:51:32.576053 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:32.576016    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-r78ll" event={"ID":"25d90578-2923-49f9-88e8-cf9f72a0ebb1","Type":"ContainerStarted","Data":"59ba94b24b6f89fbad65ac6a58f864412f92678e497416cba2020065b58ef961"}
Jul 13 18:51:34.586771 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:34.586727    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7845d98b48-vhgkl" event={"ID":"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf","Type":"ContainerStarted","Data":"338ca5d0a5eb45453d953bec75758fe4ccff4bbeab34bd4843066e324609e180"}
Jul 13 18:51:34.606723 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:34.606673    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7845d98b48-vhgkl" podStartSLOduration=1.221354175 podStartE2EDuration="4.606651807s" totalImagesPullingTime="3.385297632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:34.605892137 +0000 UTC m=+116.119691306" watchObservedRunningTime="2026-07-13 18:51:34.606651807 +0000 UTC m=+116.120450974"
Jul 13 18:51:35.595105 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:35.595067    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-r78ll" event={"ID":"25d90578-2923-49f9-88e8-cf9f72a0ebb1","Type":"ContainerStarted","Data":"b2d9dff208806f33f83dd0ea4e7ee7364d9cf18d0329c4a8ecbe46360ce01d25"}
Jul 13 18:51:35.619455 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:35.619395    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-r78ll" podStartSLOduration=3.354648975 podStartE2EDuration="5.619377203s" totalImagesPullingTime="2.264728228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:35.618090393 +0000 UTC m=+117.131889560" watchObservedRunningTime="2026-07-13 18:51:35.619377203 +0000 UTC m=+117.133176369"
Jul 13 18:51:36.599151 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:36.599116    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-r78ll"
Jul 13 18:51:36.604954 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:36.604928    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-r78ll"
Jul 13 18:51:38.015969 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:38.015937    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:51:39.537716 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.537683    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64b6bd4cb6-m2sqj"]
Jul 13 18:51:39.542249 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.542226    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:39.562663 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.562631    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 13 18:51:39.580963 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.580892    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64b6bd4cb6-m2sqj"]
Jul 13 18:51:39.581205 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.581183    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-oauth-serving-cert\") pod \"console-64b6bd4cb6-m2sqj\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") " pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:39.581329 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.581226    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/093c2706-82fd-454e-a557-0605e45034db-console-serving-cert\") pod \"console-64b6bd4cb6-m2sqj\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") " pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:39.581329 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.581250    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/093c2706-82fd-454e-a557-0605e45034db-console-oauth-config\") pod \"console-64b6bd4cb6-m2sqj\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") " pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:39.581448 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.581365    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86gnc\" (UniqueName: \"kubernetes.io/projected/093c2706-82fd-454e-a557-0605e45034db-kube-api-access-86gnc\") pod \"console-64b6bd4cb6-m2sqj\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") " pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:39.581448 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.581427    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-trusted-ca-bundle\") pod \"console-64b6bd4cb6-m2sqj\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") " pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:39.581543 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.581504    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-service-ca\") pod \"console-64b6bd4cb6-m2sqj\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") " pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:39.581595 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.581563    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-console-config\") pod \"console-64b6bd4cb6-m2sqj\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") " pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:39.682132 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.682090    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-trusted-ca-bundle\") pod \"console-64b6bd4cb6-m2sqj\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") " pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:39.682132 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.682137    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-service-ca\") pod \"console-64b6bd4cb6-m2sqj\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") " pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:39.682423 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.682177    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-console-config\") pod \"console-64b6bd4cb6-m2sqj\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") " pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:39.682423 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.682221    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-oauth-serving-cert\") pod \"console-64b6bd4cb6-m2sqj\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") " pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:39.682423 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.682254    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/093c2706-82fd-454e-a557-0605e45034db-console-serving-cert\") pod \"console-64b6bd4cb6-m2sqj\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") " pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:39.682423 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.682270    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/093c2706-82fd-454e-a557-0605e45034db-console-oauth-config\") pod \"console-64b6bd4cb6-m2sqj\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") " pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:39.682423 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.682348    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86gnc\" (UniqueName: \"kubernetes.io/projected/093c2706-82fd-454e-a557-0605e45034db-kube-api-access-86gnc\") pod \"console-64b6bd4cb6-m2sqj\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") " pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:39.682889 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.682861    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-service-ca\") pod \"console-64b6bd4cb6-m2sqj\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") " pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:39.683134 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.683109    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-trusted-ca-bundle\") pod \"console-64b6bd4cb6-m2sqj\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") " pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:39.683260 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.683238    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-console-config\") pod \"console-64b6bd4cb6-m2sqj\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") " pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:39.683799 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.683775    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-oauth-serving-cert\") pod \"console-64b6bd4cb6-m2sqj\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") " pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:39.685690 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.685668    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/093c2706-82fd-454e-a557-0605e45034db-console-oauth-config\") pod \"console-64b6bd4cb6-m2sqj\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") " pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:39.685871 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.685852    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/093c2706-82fd-454e-a557-0605e45034db-console-serving-cert\") pod \"console-64b6bd4cb6-m2sqj\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") " pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:39.693193 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.693169    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86gnc\" (UniqueName: \"kubernetes.io/projected/093c2706-82fd-454e-a557-0605e45034db-kube-api-access-86gnc\") pod \"console-64b6bd4cb6-m2sqj\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") " pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:39.852654 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:39.852615    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:40.499412 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:40.499371    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:40.499590 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:40.499445    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:40.505238 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:40.505214    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:40.618169 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:40.618136    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:51:42.931556 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:42.931513    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c"]
Jul 13 18:51:42.935142 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:42.935119    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c"
Jul 13 18:51:42.939882 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:42.939664    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 13 18:51:42.939882 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:42.939707    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 13 18:51:42.939882 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:42.939707    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 13 18:51:42.939882 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:42.939743    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 13 18:51:42.939882 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:42.939711    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 13 18:51:42.939882 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:42.939713    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4lwfn\""
Jul 13 18:51:42.955846 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:42.955819    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c"]
Jul 13 18:51:42.964693 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:42.963470    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"]
Jul 13 18:51:42.966880 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:42.966849    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"
Jul 13 18:51:42.970067 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:42.970046    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 13 18:51:42.970445 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:42.970357    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 13 18:51:42.970552 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:42.970447    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 13 18:51:42.970552 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:42.970465    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-ghwwz\""
Jul 13 18:51:42.995933 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:42.995900    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"]
Jul 13 18:51:42.998203 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:42.998183    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cw9bv"]
Jul 13 18:51:43.001752 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.001731    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.004344 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.004316    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 13 18:51:43.004444 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.004414    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zmqhc\""
Jul 13 18:51:43.004761 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.004743    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 13 18:51:43.005476 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.005461    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 13 18:51:43.012738 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.012712    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frmh9\" (UniqueName: \"kubernetes.io/projected/42789374-5d5e-4463-9db7-50101ecf2a58-kube-api-access-frmh9\") pod \"kube-state-metrics-785778ff74-8lv7g\" (UID: \"42789374-5d5e-4463-9db7-50101ecf2a58\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"
Jul 13 18:51:43.012853 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.012757    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/38462958-ea11-4e61-aad7-6154f7fc2fa8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-dmh6c\" (UID: \"38462958-ea11-4e61-aad7-6154f7fc2fa8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c"
Jul 13 18:51:43.012853 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.012792    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45c48\" (UniqueName: \"kubernetes.io/projected/38462958-ea11-4e61-aad7-6154f7fc2fa8-kube-api-access-45c48\") pod \"openshift-state-metrics-84bd68b896-dmh6c\" (UID: \"38462958-ea11-4e61-aad7-6154f7fc2fa8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c"
Jul 13 18:51:43.012853 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.012822    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42789374-5d5e-4463-9db7-50101ecf2a58-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-8lv7g\" (UID: \"42789374-5d5e-4463-9db7-50101ecf2a58\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"
Jul 13 18:51:43.012853 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.012850    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/42789374-5d5e-4463-9db7-50101ecf2a58-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-8lv7g\" (UID: \"42789374-5d5e-4463-9db7-50101ecf2a58\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"
Jul 13 18:51:43.013076 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.012883    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/42789374-5d5e-4463-9db7-50101ecf2a58-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-8lv7g\" (UID: \"42789374-5d5e-4463-9db7-50101ecf2a58\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"
Jul 13 18:51:43.013076 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.012913    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38462958-ea11-4e61-aad7-6154f7fc2fa8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-dmh6c\" (UID: \"38462958-ea11-4e61-aad7-6154f7fc2fa8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c"
Jul 13 18:51:43.013076 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.013018    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38462958-ea11-4e61-aad7-6154f7fc2fa8-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-dmh6c\" (UID: \"38462958-ea11-4e61-aad7-6154f7fc2fa8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c"
Jul 13 18:51:43.013076 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.013052    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42789374-5d5e-4463-9db7-50101ecf2a58-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-8lv7g\" (UID: \"42789374-5d5e-4463-9db7-50101ecf2a58\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"
Jul 13 18:51:43.013275 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.013096    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/42789374-5d5e-4463-9db7-50101ecf2a58-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-8lv7g\" (UID: \"42789374-5d5e-4463-9db7-50101ecf2a58\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"
Jul 13 18:51:43.113705 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.113674    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-sys\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.113705 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.113719    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frmh9\" (UniqueName: \"kubernetes.io/projected/42789374-5d5e-4463-9db7-50101ecf2a58-kube-api-access-frmh9\") pod \"kube-state-metrics-785778ff74-8lv7g\" (UID: \"42789374-5d5e-4463-9db7-50101ecf2a58\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"
Jul 13 18:51:43.113981 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.113750    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/38462958-ea11-4e61-aad7-6154f7fc2fa8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-dmh6c\" (UID: \"38462958-ea11-4e61-aad7-6154f7fc2fa8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c"
Jul 13 18:51:43.113981 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.113772    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-root\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.113981 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.113802    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.113981 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.113832    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45c48\" (UniqueName: \"kubernetes.io/projected/38462958-ea11-4e61-aad7-6154f7fc2fa8-kube-api-access-45c48\") pod \"openshift-state-metrics-84bd68b896-dmh6c\" (UID: \"38462958-ea11-4e61-aad7-6154f7fc2fa8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c"
Jul 13 18:51:43.113981 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:43.113874    2569 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 13 18:51:43.113981 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.113915    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42789374-5d5e-4463-9db7-50101ecf2a58-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-8lv7g\" (UID: \"42789374-5d5e-4463-9db7-50101ecf2a58\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"
Jul 13 18:51:43.113981 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:43.113944    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38462958-ea11-4e61-aad7-6154f7fc2fa8-openshift-state-metrics-tls podName:38462958-ea11-4e61-aad7-6154f7fc2fa8 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:43.613924727 +0000 UTC m=+125.127723876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/38462958-ea11-4e61-aad7-6154f7fc2fa8-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-dmh6c" (UID: "38462958-ea11-4e61-aad7-6154f7fc2fa8") : secret "openshift-state-metrics-tls" not found
Jul 13 18:51:43.113981 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.113975    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/42789374-5d5e-4463-9db7-50101ecf2a58-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-8lv7g\" (UID: \"42789374-5d5e-4463-9db7-50101ecf2a58\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"
Jul 13 18:51:43.114393 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.114003    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/42789374-5d5e-4463-9db7-50101ecf2a58-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-8lv7g\" (UID: \"42789374-5d5e-4463-9db7-50101ecf2a58\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"
Jul 13 18:51:43.114393 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.114035    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38462958-ea11-4e61-aad7-6154f7fc2fa8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-dmh6c\" (UID: \"38462958-ea11-4e61-aad7-6154f7fc2fa8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c"
Jul 13 18:51:43.114393 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.114090    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-node-exporter-wtmp\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.114393 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.114139    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-node-exporter-tls\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.114393 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.114253    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38462958-ea11-4e61-aad7-6154f7fc2fa8-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-dmh6c\" (UID: \"38462958-ea11-4e61-aad7-6154f7fc2fa8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c"
Jul 13 18:51:43.114393 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.114284    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42789374-5d5e-4463-9db7-50101ecf2a58-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-8lv7g\" (UID: \"42789374-5d5e-4463-9db7-50101ecf2a58\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"
Jul 13 18:51:43.114393 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.114329    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6qrz\" (UniqueName: \"kubernetes.io/projected/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-kube-api-access-k6qrz\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.114393 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.114362    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-node-exporter-textfile\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.114781 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.114503    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/42789374-5d5e-4463-9db7-50101ecf2a58-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-8lv7g\" (UID: \"42789374-5d5e-4463-9db7-50101ecf2a58\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"
Jul 13 18:51:43.114781 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.114542    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-metrics-client-ca\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.114781 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.114577    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-node-exporter-accelerators-collector-config\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.114781 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.114683    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/42789374-5d5e-4463-9db7-50101ecf2a58-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-8lv7g\" (UID: \"42789374-5d5e-4463-9db7-50101ecf2a58\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"
Jul 13 18:51:43.115088 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.115066    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38462958-ea11-4e61-aad7-6154f7fc2fa8-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-dmh6c\" (UID: \"38462958-ea11-4e61-aad7-6154f7fc2fa8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c"
Jul 13 18:51:43.115212 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.115195    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/42789374-5d5e-4463-9db7-50101ecf2a58-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-8lv7g\" (UID: \"42789374-5d5e-4463-9db7-50101ecf2a58\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"
Jul 13 18:51:43.115260 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.115238    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42789374-5d5e-4463-9db7-50101ecf2a58-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-8lv7g\" (UID: \"42789374-5d5e-4463-9db7-50101ecf2a58\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"
Jul 13 18:51:43.117141 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.117116    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38462958-ea11-4e61-aad7-6154f7fc2fa8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-dmh6c\" (UID: \"38462958-ea11-4e61-aad7-6154f7fc2fa8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c"
Jul 13 18:51:43.117234 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.117117    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42789374-5d5e-4463-9db7-50101ecf2a58-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-8lv7g\" (UID: \"42789374-5d5e-4463-9db7-50101ecf2a58\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"
Jul 13 18:51:43.117339 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.117317    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/42789374-5d5e-4463-9db7-50101ecf2a58-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-8lv7g\" (UID: \"42789374-5d5e-4463-9db7-50101ecf2a58\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"
Jul 13 18:51:43.125684 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.125663    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frmh9\" (UniqueName: \"kubernetes.io/projected/42789374-5d5e-4463-9db7-50101ecf2a58-kube-api-access-frmh9\") pod \"kube-state-metrics-785778ff74-8lv7g\" (UID: \"42789374-5d5e-4463-9db7-50101ecf2a58\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"
Jul 13 18:51:43.125799 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.125706    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45c48\" (UniqueName: \"kubernetes.io/projected/38462958-ea11-4e61-aad7-6154f7fc2fa8-kube-api-access-45c48\") pod \"openshift-state-metrics-84bd68b896-dmh6c\" (UID: \"38462958-ea11-4e61-aad7-6154f7fc2fa8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c"
Jul 13 18:51:43.215698 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.215599    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-sys\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.215698 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.215682    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-root\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.215926 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.215718    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.215926 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.215734    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-sys\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.215926 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.215779    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-node-exporter-wtmp\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.215926 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.215805    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-root\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.215926 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.215815    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-node-exporter-tls\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.215926 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.215849    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6qrz\" (UniqueName: \"kubernetes.io/projected/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-kube-api-access-k6qrz\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.215926 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.215881    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-node-exporter-textfile\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.215926 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.215909    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-metrics-client-ca\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.215926 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:43.215927    2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 13 18:51:43.216390 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.215940    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-node-exporter-accelerators-collector-config\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.216390 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.215979    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-node-exporter-wtmp\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.216390 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:43.215987    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-node-exporter-tls podName:9fcc6e8d-3e22-4ea0-955c-c631bbcbd879 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:43.715967674 +0000 UTC m=+125.229766833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-node-exporter-tls") pod "node-exporter-cw9bv" (UID: "9fcc6e8d-3e22-4ea0-955c-c631bbcbd879") : secret "node-exporter-tls" not found
Jul 13 18:51:43.216562 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.216395    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-node-exporter-textfile\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.216562 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.216532    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-node-exporter-accelerators-collector-config\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.216650 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.216568    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-metrics-client-ca\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.218812 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.218789    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.230796 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.230772    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6qrz\" (UniqueName: \"kubernetes.io/projected/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-kube-api-access-k6qrz\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.279513 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.279476    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"
Jul 13 18:51:43.620282 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.620249    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/38462958-ea11-4e61-aad7-6154f7fc2fa8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-dmh6c\" (UID: \"38462958-ea11-4e61-aad7-6154f7fc2fa8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c"
Jul 13 18:51:43.623159 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.623124    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/38462958-ea11-4e61-aad7-6154f7fc2fa8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-dmh6c\" (UID: \"38462958-ea11-4e61-aad7-6154f7fc2fa8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c"
Jul 13 18:51:43.721595 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.721559    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-node-exporter-tls\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.724181 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.724146    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fcc6e8d-3e22-4ea0-955c-c631bbcbd879-node-exporter-tls\") pod \"node-exporter-cw9bv\" (UID: \"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879\") " pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:43.846001 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.845898    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c"
Jul 13 18:51:43.913328 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:43.913227    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cw9bv"
Jul 13 18:51:45.460397 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:51:45.460246    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fcc6e8d_3e22_4ea0_955c_c631bbcbd879.slice/crio-33fd4daf352ef297aed6c36cf38138056bcd7e5af17c4e30443c95d3f4d74634 WatchSource:0}: Error finding container 33fd4daf352ef297aed6c36cf38138056bcd7e5af17c4e30443c95d3f4d74634: Status 404 returned error can't find the container with id 33fd4daf352ef297aed6c36cf38138056bcd7e5af17c4e30443c95d3f4d74634
Jul 13 18:51:45.609508 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:45.609474    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-8lv7g"]
Jul 13 18:51:45.612528 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:51:45.612499    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42789374_5d5e_4463_9db7_50101ecf2a58.slice/crio-35308959b5d57fbe65dc447902c85a9ec1b1b49b6c31b900789809073ddc3afa WatchSource:0}: Error finding container 35308959b5d57fbe65dc447902c85a9ec1b1b49b6c31b900789809073ddc3afa: Status 404 returned error can't find the container with id 35308959b5d57fbe65dc447902c85a9ec1b1b49b6c31b900789809073ddc3afa
Jul 13 18:51:45.634161 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:45.634129    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cw9bv" event={"ID":"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879","Type":"ContainerStarted","Data":"33fd4daf352ef297aed6c36cf38138056bcd7e5af17c4e30443c95d3f4d74634"}
Jul 13 18:51:45.635171 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:45.635147    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g" event={"ID":"42789374-5d5e-4463-9db7-50101ecf2a58","Type":"ContainerStarted","Data":"35308959b5d57fbe65dc447902c85a9ec1b1b49b6c31b900789809073ddc3afa"}
Jul 13 18:51:45.838327 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:45.838229    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c"]
Jul 13 18:51:45.842262 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:51:45.842230    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38462958_ea11_4e61_aad7_6154f7fc2fa8.slice/crio-16d0a929341a24507fe48497ed68ca72f19a117a40f503d8a4feb48a32b351eb WatchSource:0}: Error finding container 16d0a929341a24507fe48497ed68ca72f19a117a40f503d8a4feb48a32b351eb: Status 404 returned error can't find the container with id 16d0a929341a24507fe48497ed68ca72f19a117a40f503d8a4feb48a32b351eb
Jul 13 18:51:45.845775 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:45.845739    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64b6bd4cb6-m2sqj"]
Jul 13 18:51:46.643226 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:46.643186    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-9tnzg" event={"ID":"92c2bd01-a76e-476c-b074-3a33fd725f90","Type":"ContainerStarted","Data":"c88f36123580ad428c50b1cd9f8b44d9c10a260599aa0c29a71ae17b9efaf85d"}
Jul 13 18:51:46.643721 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:46.643417    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-9tnzg"
Jul 13 18:51:46.646881 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:46.646844    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c" event={"ID":"38462958-ea11-4e61-aad7-6154f7fc2fa8","Type":"ContainerStarted","Data":"51fc0308822e25dd1c0acfd33c920d9cc5d5f6ccd6cda8b8cdaf63135833e3ef"}
Jul 13 18:51:46.647028 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:46.646911    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c" event={"ID":"38462958-ea11-4e61-aad7-6154f7fc2fa8","Type":"ContainerStarted","Data":"f9540d09dd6788b42012c9e81c9ff04693e527972a0219407219d3451c9e680d"}
Jul 13 18:51:46.647028 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:46.646937    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c" event={"ID":"38462958-ea11-4e61-aad7-6154f7fc2fa8","Type":"ContainerStarted","Data":"16d0a929341a24507fe48497ed68ca72f19a117a40f503d8a4feb48a32b351eb"}
Jul 13 18:51:46.648828 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:46.648801    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64b6bd4cb6-m2sqj" event={"ID":"093c2706-82fd-454e-a557-0605e45034db","Type":"ContainerStarted","Data":"faa039e77b0cadfb66926d0ba316f152b63a3f98048a1b7794ba6aac5a2c7fdf"}
Jul 13 18:51:46.648996 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:46.648848    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64b6bd4cb6-m2sqj" event={"ID":"093c2706-82fd-454e-a557-0605e45034db","Type":"ContainerStarted","Data":"a91b8cad75e4c226042e81f796e2cb99b7481d01e98375f1c0f7faa6e56a26f0"}
Jul 13 18:51:46.657212 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:46.657188    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-9tnzg"
Jul 13 18:51:46.709129 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:46.709063    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-9tnzg" podStartSLOduration=2.5505923360000002 podStartE2EDuration="19.709042352s" totalImagesPullingTime="17.158450016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:46.675606536 +0000 UTC m=+128.189405704" watchObservedRunningTime="2026-07-13 18:51:46.709042352 +0000 UTC m=+128.222841569"
Jul 13 18:51:46.709774 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:46.709736    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64b6bd4cb6-m2sqj" podStartSLOduration=7.7097248480000005 podStartE2EDuration="7.709724848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:46.708124002 +0000 UTC m=+128.221923169" watchObservedRunningTime="2026-07-13 18:51:46.709724848 +0000 UTC m=+128.223524013"
Jul 13 18:51:47.379999 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.379959    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-ffb8ff984-rft4x"]
Jul 13 18:51:47.398159 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.398107    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-ffb8ff984-rft4x"]
Jul 13 18:51:47.398355 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.398238    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:47.403206 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.403180    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 13 18:51:47.403535 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.403462    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 13 18:51:47.403683 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.403655    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3dk0mqfnfi3bs\""
Jul 13 18:51:47.403921 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.403851    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 13 18:51:47.404102 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.404061    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hdhlr\""
Jul 13 18:51:47.404327 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.404309    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 13 18:51:47.458600 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.458558    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d5e724d7-11cf-4346-8295-ef996b0eb651-secret-metrics-server-tls\") pod \"metrics-server-ffb8ff984-rft4x\" (UID: \"d5e724d7-11cf-4346-8295-ef996b0eb651\") " pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:47.458818 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.458631    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d5e724d7-11cf-4346-8295-ef996b0eb651-secret-metrics-server-client-certs\") pod \"metrics-server-ffb8ff984-rft4x\" (UID: \"d5e724d7-11cf-4346-8295-ef996b0eb651\") " pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:47.458818 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.458686    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bkfj\" (UniqueName: \"kubernetes.io/projected/d5e724d7-11cf-4346-8295-ef996b0eb651-kube-api-access-9bkfj\") pod \"metrics-server-ffb8ff984-rft4x\" (UID: \"d5e724d7-11cf-4346-8295-ef996b0eb651\") " pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:47.458818 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.458779    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5e724d7-11cf-4346-8295-ef996b0eb651-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-ffb8ff984-rft4x\" (UID: \"d5e724d7-11cf-4346-8295-ef996b0eb651\") " pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:47.458974 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.458822    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d5e724d7-11cf-4346-8295-ef996b0eb651-metrics-server-audit-profiles\") pod \"metrics-server-ffb8ff984-rft4x\" (UID: \"d5e724d7-11cf-4346-8295-ef996b0eb651\") " pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:47.458974 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.458877    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d5e724d7-11cf-4346-8295-ef996b0eb651-audit-log\") pod \"metrics-server-ffb8ff984-rft4x\" (UID: \"d5e724d7-11cf-4346-8295-ef996b0eb651\") " pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:47.458974 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.458919    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d5e724d7-11cf-4346-8295-ef996b0eb651-client-ca-bundle\") pod \"metrics-server-ffb8ff984-rft4x\" (UID: \"d5e724d7-11cf-4346-8295-ef996b0eb651\") " pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:47.560392 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.560351    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d5e724d7-11cf-4346-8295-ef996b0eb651-secret-metrics-server-client-certs\") pod \"metrics-server-ffb8ff984-rft4x\" (UID: \"d5e724d7-11cf-4346-8295-ef996b0eb651\") " pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:47.560572 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.560434    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bkfj\" (UniqueName: \"kubernetes.io/projected/d5e724d7-11cf-4346-8295-ef996b0eb651-kube-api-access-9bkfj\") pod \"metrics-server-ffb8ff984-rft4x\" (UID: \"d5e724d7-11cf-4346-8295-ef996b0eb651\") " pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:47.560572 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.560492    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5e724d7-11cf-4346-8295-ef996b0eb651-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-ffb8ff984-rft4x\" (UID: \"d5e724d7-11cf-4346-8295-ef996b0eb651\") " pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:47.560572 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.560520    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d5e724d7-11cf-4346-8295-ef996b0eb651-metrics-server-audit-profiles\") pod \"metrics-server-ffb8ff984-rft4x\" (UID: \"d5e724d7-11cf-4346-8295-ef996b0eb651\") " pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:47.560572 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.560567    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d5e724d7-11cf-4346-8295-ef996b0eb651-audit-log\") pod \"metrics-server-ffb8ff984-rft4x\" (UID: \"d5e724d7-11cf-4346-8295-ef996b0eb651\") " pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:47.560780 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.560620    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d5e724d7-11cf-4346-8295-ef996b0eb651-client-ca-bundle\") pod \"metrics-server-ffb8ff984-rft4x\" (UID: \"d5e724d7-11cf-4346-8295-ef996b0eb651\") " pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:47.560780 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.560676    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d5e724d7-11cf-4346-8295-ef996b0eb651-secret-metrics-server-tls\") pod \"metrics-server-ffb8ff984-rft4x\" (UID: \"d5e724d7-11cf-4346-8295-ef996b0eb651\") " pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:47.561478 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.561424    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d5e724d7-11cf-4346-8295-ef996b0eb651-audit-log\") pod \"metrics-server-ffb8ff984-rft4x\" (UID: \"d5e724d7-11cf-4346-8295-ef996b0eb651\") " pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:47.561713 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.561686    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5e724d7-11cf-4346-8295-ef996b0eb651-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-ffb8ff984-rft4x\" (UID: \"d5e724d7-11cf-4346-8295-ef996b0eb651\") " pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:47.561713 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.562115    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d5e724d7-11cf-4346-8295-ef996b0eb651-metrics-server-audit-profiles\") pod \"metrics-server-ffb8ff984-rft4x\" (UID: \"d5e724d7-11cf-4346-8295-ef996b0eb651\") " pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:47.564985 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.564960    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d5e724d7-11cf-4346-8295-ef996b0eb651-client-ca-bundle\") pod \"metrics-server-ffb8ff984-rft4x\" (UID: \"d5e724d7-11cf-4346-8295-ef996b0eb651\") " pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:47.565509 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.565459    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d5e724d7-11cf-4346-8295-ef996b0eb651-secret-metrics-server-client-certs\") pod \"metrics-server-ffb8ff984-rft4x\" (UID: \"d5e724d7-11cf-4346-8295-ef996b0eb651\") " pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:47.569093 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.569044    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d5e724d7-11cf-4346-8295-ef996b0eb651-secret-metrics-server-tls\") pod \"metrics-server-ffb8ff984-rft4x\" (UID: \"d5e724d7-11cf-4346-8295-ef996b0eb651\") " pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:47.571854 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.571809    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bkfj\" (UniqueName: \"kubernetes.io/projected/d5e724d7-11cf-4346-8295-ef996b0eb651-kube-api-access-9bkfj\") pod \"metrics-server-ffb8ff984-rft4x\" (UID: \"d5e724d7-11cf-4346-8295-ef996b0eb651\") " pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:47.712542 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:47.712457    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:51:48.346984 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:48.346896    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-ffb8ff984-rft4x"]
Jul 13 18:51:48.350004 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:51:48.349973    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e724d7_11cf_4346_8295_ef996b0eb651.slice/crio-c487737d1b46f9bf28183ce06f682046808494ff1303b29dba8c25eaeb33f675 WatchSource:0}: Error finding container c487737d1b46f9bf28183ce06f682046808494ff1303b29dba8c25eaeb33f675: Status 404 returned error can't find the container with id c487737d1b46f9bf28183ce06f682046808494ff1303b29dba8c25eaeb33f675
Jul 13 18:51:48.659693 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:48.659614    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x" event={"ID":"d5e724d7-11cf-4346-8295-ef996b0eb651","Type":"ContainerStarted","Data":"c487737d1b46f9bf28183ce06f682046808494ff1303b29dba8c25eaeb33f675"}
Jul 13 18:51:48.662381 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:48.662311    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c" event={"ID":"38462958-ea11-4e61-aad7-6154f7fc2fa8","Type":"ContainerStarted","Data":"6e3dcf4522c05cb01e721f246d8c36a1e7ade75324a4a720645bc9899a9f71e4"}
Jul 13 18:51:48.664557 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:48.664530    2569 generic.go:356] "Generic (PLEG): container finished" podID="9fcc6e8d-3e22-4ea0-955c-c631bbcbd879" containerID="103f8ea09be48c99595f81eff8a05778e357a97ed083a759e88405e0712bf71f" exitCode=0
Jul 13 18:51:48.664941 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:48.664784    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cw9bv" event={"ID":"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879","Type":"ContainerDied","Data":"103f8ea09be48c99595f81eff8a05778e357a97ed083a759e88405e0712bf71f"}
Jul 13 18:51:48.667420 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:48.667387    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g" event={"ID":"42789374-5d5e-4463-9db7-50101ecf2a58","Type":"ContainerStarted","Data":"ec75951e1de33ba2b498e825ae8fa0016e73b693aede478bf42cb47741ce2b9d"}
Jul 13 18:51:48.667530 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:48.667459    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g" event={"ID":"42789374-5d5e-4463-9db7-50101ecf2a58","Type":"ContainerStarted","Data":"36992f839b16fd4c1dceb9109b654ebaeb8a86d51596ac1669d397d227c2120d"}
Jul 13 18:51:48.667530 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:48.667491    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g" event={"ID":"42789374-5d5e-4463-9db7-50101ecf2a58","Type":"ContainerStarted","Data":"233077db27ee25f2afd36fe0a18fc9b8f45d9695705bedbcf101ee3ed89b39fd"}
Jul 13 18:51:48.723372 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:48.723315    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-8lv7g" podStartSLOduration=4.164135831 podStartE2EDuration="6.72326895s" totalImagesPullingTime="2.559133119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:48.72318576 +0000 UTC m=+130.236984927" watchObservedRunningTime="2026-07-13 18:51:48.72326895 +0000 UTC m=+130.237068117"
Jul 13 18:51:48.724181 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:48.724149    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dmh6c" podStartSLOduration=4.58210479 podStartE2EDuration="6.724138715s" totalImagesPullingTime="2.142033925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:48.695805057 +0000 UTC m=+130.209604224" watchObservedRunningTime="2026-07-13 18:51:48.724138715 +0000 UTC m=+130.237937881"
Jul 13 18:51:48.874272 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:48.874240    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs\") pod \"network-metrics-daemon-xnfv2\" (UID: \"6ec3b9c9-8324-4d12-92b8-6baa78773fe2\") " pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:51:48.877448 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:48.877418    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec3b9c9-8324-4d12-92b8-6baa78773fe2-metrics-certs\") pod \"network-metrics-daemon-xnfv2\" (UID: \"6ec3b9c9-8324-4d12-92b8-6baa78773fe2\") " pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:51:49.170650 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.170453    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gsq9l\""
Jul 13 18:51:49.178566 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.178486    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnfv2"
Jul 13 18:51:49.259262 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.259227    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 18:51:49.283764 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.283738    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.289480 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.289180    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pctbk\""
Jul 13 18:51:49.290991 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.290931    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 13 18:51:49.291223 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.291128    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 13 18:51:49.291345 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.291329    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 13 18:51:49.291552 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.291531    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 13 18:51:49.291744 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.291728    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 13 18:51:49.297745 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.297176    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 13 18:51:49.297745 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.297453    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 13 18:51:49.297745 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.297678    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 13 18:51:49.297951 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.297899    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 13 18:51:49.298598 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.298075    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 13 18:51:49.298598 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.298175    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 13 18:51:49.298598 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.298223    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 13 18:51:49.298598 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.298270    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-88659namsui2e\""
Jul 13 18:51:49.298598 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.298397    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 13 18:51:49.326767 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.322218    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 18:51:49.370504 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.370466    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 13 18:51:49.381205 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.381171    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.381553 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.381532    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.381758 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.381740    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.381890 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.381879    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.382124 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.381983    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.382784 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.382258    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-config-out\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.383584 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.383558    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.383679 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.383604    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-config\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.383679 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.383638    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htdn5\" (UniqueName: \"kubernetes.io/projected/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-kube-api-access-htdn5\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.383782 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.383686    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.383782 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.383711    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.383782 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.383735    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.383782 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.383757    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.383975 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.383795    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.383975 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.383832    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.383975 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.383863    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-web-config\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.383975 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.383892    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.383975 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.383917    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.383975 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.383942    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.383975 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.383965    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.446538 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.446026    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xnfv2"]
Jul 13 18:51:49.449652 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:51:49.449619    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec3b9c9_8324_4d12_92b8_6baa78773fe2.slice/crio-c24634b3d22ed6890532e8638dfcd3adcc1f8b1c1e82b4e89280efa7adf49be2 WatchSource:0}: Error finding container c24634b3d22ed6890532e8638dfcd3adcc1f8b1c1e82b4e89280efa7adf49be2: Status 404 returned error can't find the container with id c24634b3d22ed6890532e8638dfcd3adcc1f8b1c1e82b4e89280efa7adf49be2
Jul 13 18:51:49.485368 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.485322    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htdn5\" (UniqueName: \"kubernetes.io/projected/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-kube-api-access-htdn5\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.485561 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.485392    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.485561 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.485415    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.485561 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.485436    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.485561 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.485457    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.485561 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.485495    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.485561 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.485527    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.485561 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.485553    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-web-config\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.485996 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.485580    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.485996 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.485604    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.485996 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.485629    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.485996 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.485687    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.485996 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.485754    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.485996 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.485817    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.485996 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.485845    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.485996 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.485884    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.485996 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.485917    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.485996 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.485951    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-config-out\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.485996 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.485975    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.486529 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.486004    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-config\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.486740 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.486718    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.493757 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.487117    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.493757 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.489708    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-web-config\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.493757 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.489807    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.493757 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.490924    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.493757 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.491820    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.493757 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.492342    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.493757 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.492511    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.493757 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.492892    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-config-out\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.493757 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.493142    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.496428 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.494673    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.496428 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.495808    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.496859 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.496823    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.497828 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.497348    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.497828 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.497744    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.497828 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.497787    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.498010 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.497864    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.498459 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.498170    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.498459 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.498424    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-config\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.508462 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.508431    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htdn5\" (UniqueName: \"kubernetes.io/projected/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-kube-api-access-htdn5\") pod \"prometheus-k8s-0\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.603154 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.603113    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:49.675124 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.674703    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cw9bv" event={"ID":"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879","Type":"ContainerStarted","Data":"ec711c95fd0b7401255ac612658a8c6984c57aed25fff81762e2ce4415b2afb0"}
Jul 13 18:51:49.675124 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.674769    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cw9bv" event={"ID":"9fcc6e8d-3e22-4ea0-955c-c631bbcbd879","Type":"ContainerStarted","Data":"7b661ad10e0651c1c2a822afea30722e9b54caa89735987d0def6f4c2e804e62"}
Jul 13 18:51:49.676380 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.676350    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xnfv2" event={"ID":"6ec3b9c9-8324-4d12-92b8-6baa78773fe2","Type":"ContainerStarted","Data":"c24634b3d22ed6890532e8638dfcd3adcc1f8b1c1e82b4e89280efa7adf49be2"}
Jul 13 18:51:49.727957 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.727495    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cw9bv" podStartSLOduration=4.9210443680000004 podStartE2EDuration="7.727473075s" totalImagesPullingTime="2.680290124s" totalInitContainerRuntime="126.138583ms" isStatefulPod=true podCreationTimestamp="2026-07-13 18:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:49.725865489 +0000 UTC m=+131.239664656" watchObservedRunningTime="2026-07-13 18:51:49.727473075 +0000 UTC m=+131.241272242"
Jul 13 18:51:49.820522 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.820485    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 18:51:49.852923 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.852889    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:49.853891 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.853809    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:49.862335 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:49.862311    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:50.188310 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:51:50.188258    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe8c10d_cc0c_4133_bffb_973b4a57a6c8.slice/crio-d00709f722b69671b4e3f06969f3928c86909d6a5ff181c489bc165124699455 WatchSource:0}: Error finding container d00709f722b69671b4e3f06969f3928c86909d6a5ff181c489bc165124699455: Status 404 returned error can't find the container with id d00709f722b69671b4e3f06969f3928c86909d6a5ff181c489bc165124699455
Jul 13 18:51:50.684881 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:50.684833    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x" event={"ID":"d5e724d7-11cf-4346-8295-ef996b0eb651","Type":"ContainerStarted","Data":"e470bf919051331215848920c5a6883d4deb7b1aaa9f5bc2574d1c5c371ef3e9"}
Jul 13 18:51:50.687545 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:50.687514    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8","Type":"ContainerStarted","Data":"d00709f722b69671b4e3f06969f3928c86909d6a5ff181c489bc165124699455"}
Jul 13 18:51:50.697912 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:50.697741    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:51:50.752319 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:50.751562    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7845d98b48-vhgkl"]
Jul 13 18:51:52.698092 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:52.698057    2569 generic.go:356] "Generic (PLEG): container finished" podID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerID="2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366" exitCode=0
Jul 13 18:51:52.698527 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:52.698155    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8","Type":"ContainerDied","Data":"2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366"}
Jul 13 18:51:52.701117 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:52.701038    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xnfv2" event={"ID":"6ec3b9c9-8324-4d12-92b8-6baa78773fe2","Type":"ContainerStarted","Data":"49657690ed23458015f366ad720e5c90ca891abae295e3ab4caf1a131dc1edbc"}
Jul 13 18:51:52.748223 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:52.747946    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x" podStartSLOduration=3.554100536 podStartE2EDuration="5.747924375s" totalImagesPullingTime="2.193823839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:51.713155912 +0000 UTC m=+133.226955081" watchObservedRunningTime="2026-07-13 18:51:52.747924375 +0000 UTC m=+134.261723541"
Jul 13 18:51:53.032262 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.032145    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64f848497f-8c2qx" podUID="73a82b38-5c25-40f0-9849-4b0be568ae02" containerName="registry" containerID="cri-o://b32f9f2b0b9d5075f220f82e05fd6b4cf75138fc9326bc3b1760e2010cf09712" gracePeriod=30
Jul 13 18:51:53.343267 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.343188    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:51:53.434690 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.433564    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73a82b38-5c25-40f0-9849-4b0be568ae02-trusted-ca\") pod \"73a82b38-5c25-40f0-9849-4b0be568ae02\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") "
Jul 13 18:51:53.434690 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.433622    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhklf\" (UniqueName: \"kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-kube-api-access-lhklf\") pod \"73a82b38-5c25-40f0-9849-4b0be568ae02\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") "
Jul 13 18:51:53.434690 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.433703    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73a82b38-5c25-40f0-9849-4b0be568ae02-ca-trust-extracted\") pod \"73a82b38-5c25-40f0-9849-4b0be568ae02\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") "
Jul 13 18:51:53.434690 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.433738    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73a82b38-5c25-40f0-9849-4b0be568ae02-image-registry-private-configuration\") pod \"73a82b38-5c25-40f0-9849-4b0be568ae02\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") "
Jul 13 18:51:53.434690 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.433781    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-tls\") pod \"73a82b38-5c25-40f0-9849-4b0be568ae02\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") "
Jul 13 18:51:53.434690 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.433824    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-bound-sa-token\") pod \"73a82b38-5c25-40f0-9849-4b0be568ae02\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") "
Jul 13 18:51:53.434690 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.433861    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-certificates\") pod \"73a82b38-5c25-40f0-9849-4b0be568ae02\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") "
Jul 13 18:51:53.434690 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.433891    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73a82b38-5c25-40f0-9849-4b0be568ae02-installation-pull-secrets\") pod \"73a82b38-5c25-40f0-9849-4b0be568ae02\" (UID: \"73a82b38-5c25-40f0-9849-4b0be568ae02\") "
Jul 13 18:51:53.436127 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.435989    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a82b38-5c25-40f0-9849-4b0be568ae02-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "73a82b38-5c25-40f0-9849-4b0be568ae02" (UID: "73a82b38-5c25-40f0-9849-4b0be568ae02"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:51:53.436362 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.436184    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "73a82b38-5c25-40f0-9849-4b0be568ae02" (UID: "73a82b38-5c25-40f0-9849-4b0be568ae02"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:51:53.438338 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.438266    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73a82b38-5c25-40f0-9849-4b0be568ae02-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "73a82b38-5c25-40f0-9849-4b0be568ae02" (UID: "73a82b38-5c25-40f0-9849-4b0be568ae02"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:51:53.444595 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.444561    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-kube-api-access-lhklf" (OuterVolumeSpecName: "kube-api-access-lhklf") pod "73a82b38-5c25-40f0-9849-4b0be568ae02" (UID: "73a82b38-5c25-40f0-9849-4b0be568ae02"). InnerVolumeSpecName "kube-api-access-lhklf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:51:53.444887 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.444789    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73a82b38-5c25-40f0-9849-4b0be568ae02-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "73a82b38-5c25-40f0-9849-4b0be568ae02" (UID: "73a82b38-5c25-40f0-9849-4b0be568ae02"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:51:53.445577 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.445150    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "73a82b38-5c25-40f0-9849-4b0be568ae02" (UID: "73a82b38-5c25-40f0-9849-4b0be568ae02"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:51:53.447021 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.446984    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "73a82b38-5c25-40f0-9849-4b0be568ae02" (UID: "73a82b38-5c25-40f0-9849-4b0be568ae02"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:51:53.449208 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.449162    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a82b38-5c25-40f0-9849-4b0be568ae02-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "73a82b38-5c25-40f0-9849-4b0be568ae02" (UID: "73a82b38-5c25-40f0-9849-4b0be568ae02"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:51:53.534924 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.534880    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhklf\" (UniqueName: \"kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-kube-api-access-lhklf\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:51:53.534924 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.534921    2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73a82b38-5c25-40f0-9849-4b0be568ae02-ca-trust-extracted\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:51:53.535174 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.534935    2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73a82b38-5c25-40f0-9849-4b0be568ae02-image-registry-private-configuration\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:51:53.535174 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.534951    2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-tls\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:51:53.535174 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.534964    2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73a82b38-5c25-40f0-9849-4b0be568ae02-bound-sa-token\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:51:53.535174 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.534975    2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73a82b38-5c25-40f0-9849-4b0be568ae02-registry-certificates\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:51:53.535174 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.534989    2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73a82b38-5c25-40f0-9849-4b0be568ae02-installation-pull-secrets\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:51:53.535174 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.535002    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73a82b38-5c25-40f0-9849-4b0be568ae02-trusted-ca\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:51:53.719708 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.719659    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xnfv2" event={"ID":"6ec3b9c9-8324-4d12-92b8-6baa78773fe2","Type":"ContainerStarted","Data":"108a0ad07835c298cb27bdf2aafc5d733656ea2b2f45868c6a5d831417a1f752"}
Jul 13 18:51:53.721698 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.721668    2569 generic.go:356] "Generic (PLEG): container finished" podID="73a82b38-5c25-40f0-9849-4b0be568ae02" containerID="b32f9f2b0b9d5075f220f82e05fd6b4cf75138fc9326bc3b1760e2010cf09712" exitCode=0
Jul 13 18:51:53.721944 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.721925    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64f848497f-8c2qx" event={"ID":"73a82b38-5c25-40f0-9849-4b0be568ae02","Type":"ContainerDied","Data":"b32f9f2b0b9d5075f220f82e05fd6b4cf75138fc9326bc3b1760e2010cf09712"}
Jul 13 18:51:53.722062 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.722048    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64f848497f-8c2qx" event={"ID":"73a82b38-5c25-40f0-9849-4b0be568ae02","Type":"ContainerDied","Data":"3c4c8bf6f74e7052a10044b1d2e55ce0646cb61c140c51d1ba524cd67416528c"}
Jul 13 18:51:53.722164 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.722152    2569 scope.go:117] "RemoveContainer" containerID="b32f9f2b0b9d5075f220f82e05fd6b4cf75138fc9326bc3b1760e2010cf09712"
Jul 13 18:51:53.722355 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.722332    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64f848497f-8c2qx"
Jul 13 18:51:53.740375 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.740203    2569 scope.go:117] "RemoveContainer" containerID="b32f9f2b0b9d5075f220f82e05fd6b4cf75138fc9326bc3b1760e2010cf09712"
Jul 13 18:51:53.741241 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.741192    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xnfv2" podStartSLOduration=131.771533608 podStartE2EDuration="2m14.741177998s" totalImagesPullingTime="2.96964439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:53.739872215 +0000 UTC m=+135.253671382" watchObservedRunningTime="2026-07-13 18:51:53.741177998 +0000 UTC m=+135.254977165"
Jul 13 18:51:53.741431 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:51:53.741401    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b32f9f2b0b9d5075f220f82e05fd6b4cf75138fc9326bc3b1760e2010cf09712\": container with ID starting with b32f9f2b0b9d5075f220f82e05fd6b4cf75138fc9326bc3b1760e2010cf09712 not found: ID does not exist" containerID="b32f9f2b0b9d5075f220f82e05fd6b4cf75138fc9326bc3b1760e2010cf09712"
Jul 13 18:51:53.741503 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.741447    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b32f9f2b0b9d5075f220f82e05fd6b4cf75138fc9326bc3b1760e2010cf09712"} err="failed to get container status \"b32f9f2b0b9d5075f220f82e05fd6b4cf75138fc9326bc3b1760e2010cf09712\": rpc error: code = NotFound desc = could not find container \"b32f9f2b0b9d5075f220f82e05fd6b4cf75138fc9326bc3b1760e2010cf09712\": container with ID starting with b32f9f2b0b9d5075f220f82e05fd6b4cf75138fc9326bc3b1760e2010cf09712 not found: ID does not exist"
Jul 13 18:51:53.763375 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.763314    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64f848497f-8c2qx"]
Jul 13 18:51:53.768642 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:53.768610    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64f848497f-8c2qx"]
Jul 13 18:51:55.061355 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:55.061316    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73a82b38-5c25-40f0-9849-4b0be568ae02" path="/var/lib/kubelet/pods/73a82b38-5c25-40f0-9849-4b0be568ae02/volumes"
Jul 13 18:51:56.740071 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:56.740000    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8","Type":"ContainerStarted","Data":"abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc"}
Jul 13 18:51:56.740071 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:51:56.740066    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8","Type":"ContainerStarted","Data":"6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd"}
Jul 13 18:52:02.692484 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:02.692446    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64b6bd4cb6-m2sqj"]
Jul 13 18:52:02.770843 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:02.770806    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8","Type":"ContainerStarted","Data":"16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50"}
Jul 13 18:52:02.771006 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:02.770888    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8","Type":"ContainerStarted","Data":"0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76"}
Jul 13 18:52:02.771006 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:02.770928    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8","Type":"ContainerStarted","Data":"d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96"}
Jul 13 18:52:02.771006 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:02.770973    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8","Type":"ContainerStarted","Data":"3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c"}
Jul 13 18:52:02.810903 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:02.810850    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.436886441 podStartE2EDuration="13.810833004s" totalImagesPullingTime="11.345778559s" totalInitContainerRuntime="28.168004ms" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:02.808758802 +0000 UTC m=+144.322557967" watchObservedRunningTime="2026-07-13 18:52:02.810833004 +0000 UTC m=+144.324632173"
Jul 13 18:52:04.604133 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:04.604096    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:07.713120 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:07.713088    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:52:07.713519 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:07.713157    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:52:14.823404 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:14.823371    2569 generic.go:356] "Generic (PLEG): container finished" podID="7bd176fb-f16d-4c0b-99ac-ef0bb25b1390" containerID="060009844f402070d4269afcf28e4b77f51fe4c1befc3ab34cf65826655f106d" exitCode=0
Jul 13 18:52:14.823822 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:14.823442    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-9pt94" event={"ID":"7bd176fb-f16d-4c0b-99ac-ef0bb25b1390","Type":"ContainerDied","Data":"060009844f402070d4269afcf28e4b77f51fe4c1befc3ab34cf65826655f106d"}
Jul 13 18:52:14.823822 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:14.823760    2569 scope.go:117] "RemoveContainer" containerID="060009844f402070d4269afcf28e4b77f51fe4c1befc3ab34cf65826655f106d"
Jul 13 18:52:15.799927 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:15.799866    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7845d98b48-vhgkl" podUID="668d7b34-95bb-46ab-a0ba-1d5dd4911ddf" containerName="console" containerID="cri-o://338ca5d0a5eb45453d953bec75758fe4ccff4bbeab34bd4843066e324609e180" gracePeriod=15
Jul 13 18:52:15.829497 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:15.829466    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-9pt94" event={"ID":"7bd176fb-f16d-4c0b-99ac-ef0bb25b1390","Type":"ContainerStarted","Data":"c8818a6903ab9ad9862e517d9dd2d535bc7bf4a46edfeb0add173a7685cccceb"}
Jul 13 18:52:16.050667 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.050604    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:52:16.117088 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.117049    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-oauth-serving-cert\") pod \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") "
Jul 13 18:52:16.117279 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.117109    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-console-serving-cert\") pod \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") "
Jul 13 18:52:16.117279 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.117142    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-service-ca\") pod \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") "
Jul 13 18:52:16.117279 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.117174    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv6bx\" (UniqueName: \"kubernetes.io/projected/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-kube-api-access-sv6bx\") pod \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") "
Jul 13 18:52:16.117279 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.117201    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-console-oauth-config\") pod \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") "
Jul 13 18:52:16.117279 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.117235    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-console-config\") pod \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\" (UID: \"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf\") "
Jul 13 18:52:16.117624 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.117562    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "668d7b34-95bb-46ab-a0ba-1d5dd4911ddf" (UID: "668d7b34-95bb-46ab-a0ba-1d5dd4911ddf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:52:16.117693 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.117655    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-service-ca" (OuterVolumeSpecName: "service-ca") pod "668d7b34-95bb-46ab-a0ba-1d5dd4911ddf" (UID: "668d7b34-95bb-46ab-a0ba-1d5dd4911ddf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:52:16.117898 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.117872    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-console-config" (OuterVolumeSpecName: "console-config") pod "668d7b34-95bb-46ab-a0ba-1d5dd4911ddf" (UID: "668d7b34-95bb-46ab-a0ba-1d5dd4911ddf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:52:16.119759 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.119733    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-kube-api-access-sv6bx" (OuterVolumeSpecName: "kube-api-access-sv6bx") pod "668d7b34-95bb-46ab-a0ba-1d5dd4911ddf" (UID: "668d7b34-95bb-46ab-a0ba-1d5dd4911ddf"). InnerVolumeSpecName "kube-api-access-sv6bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:52:16.119855 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.119754    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "668d7b34-95bb-46ab-a0ba-1d5dd4911ddf" (UID: "668d7b34-95bb-46ab-a0ba-1d5dd4911ddf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:52:16.119855 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.119795    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "668d7b34-95bb-46ab-a0ba-1d5dd4911ddf" (UID: "668d7b34-95bb-46ab-a0ba-1d5dd4911ddf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:52:16.218401 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.218350    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-console-oauth-config\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:52:16.218401 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.218396    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-console-config\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:52:16.218401 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.218410    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-oauth-serving-cert\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:52:16.218640 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.218424    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-console-serving-cert\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:52:16.218640 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.218437    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-service-ca\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:52:16.218640 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.218449    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv6bx\" (UniqueName: \"kubernetes.io/projected/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf-kube-api-access-sv6bx\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:52:16.834049 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.834018    2569 generic.go:356] "Generic (PLEG): container finished" podID="668d7b34-95bb-46ab-a0ba-1d5dd4911ddf" containerID="338ca5d0a5eb45453d953bec75758fe4ccff4bbeab34bd4843066e324609e180" exitCode=0
Jul 13 18:52:16.834446 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.834061    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7845d98b48-vhgkl"
Jul 13 18:52:16.834446 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.834108    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7845d98b48-vhgkl" event={"ID":"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf","Type":"ContainerDied","Data":"338ca5d0a5eb45453d953bec75758fe4ccff4bbeab34bd4843066e324609e180"}
Jul 13 18:52:16.834446 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.834158    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7845d98b48-vhgkl" event={"ID":"668d7b34-95bb-46ab-a0ba-1d5dd4911ddf","Type":"ContainerDied","Data":"b1d18b48a22a6daf799a2c4d445557d2289e2a187f3117805c2ff5084e32242f"}
Jul 13 18:52:16.834446 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.834183    2569 scope.go:117] "RemoveContainer" containerID="338ca5d0a5eb45453d953bec75758fe4ccff4bbeab34bd4843066e324609e180"
Jul 13 18:52:16.842864 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.842848    2569 scope.go:117] "RemoveContainer" containerID="338ca5d0a5eb45453d953bec75758fe4ccff4bbeab34bd4843066e324609e180"
Jul 13 18:52:16.843099 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:52:16.843079    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"338ca5d0a5eb45453d953bec75758fe4ccff4bbeab34bd4843066e324609e180\": container with ID starting with 338ca5d0a5eb45453d953bec75758fe4ccff4bbeab34bd4843066e324609e180 not found: ID does not exist" containerID="338ca5d0a5eb45453d953bec75758fe4ccff4bbeab34bd4843066e324609e180"
Jul 13 18:52:16.843150 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.843107    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338ca5d0a5eb45453d953bec75758fe4ccff4bbeab34bd4843066e324609e180"} err="failed to get container status \"338ca5d0a5eb45453d953bec75758fe4ccff4bbeab34bd4843066e324609e180\": rpc error: code = NotFound desc = could not find container \"338ca5d0a5eb45453d953bec75758fe4ccff4bbeab34bd4843066e324609e180\": container with ID starting with 338ca5d0a5eb45453d953bec75758fe4ccff4bbeab34bd4843066e324609e180 not found: ID does not exist"
Jul 13 18:52:16.854806 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.854780    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7845d98b48-vhgkl"]
Jul 13 18:52:16.858304 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:16.858271    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7845d98b48-vhgkl"]
Jul 13 18:52:17.060171 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:17.060139    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="668d7b34-95bb-46ab-a0ba-1d5dd4911ddf" path="/var/lib/kubelet/pods/668d7b34-95bb-46ab-a0ba-1d5dd4911ddf/volumes"
Jul 13 18:52:19.851081 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:19.851051    2569 generic.go:356] "Generic (PLEG): container finished" podID="8977c40b-f5c4-4aea-a441-6611214b30f7" containerID="1f99bf2136fc57e74e231a5ba5a0423b67ec05c34283fcc048306503b5a15415" exitCode=0
Jul 13 18:52:19.851532 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:19.851130    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd" event={"ID":"8977c40b-f5c4-4aea-a441-6611214b30f7","Type":"ContainerDied","Data":"1f99bf2136fc57e74e231a5ba5a0423b67ec05c34283fcc048306503b5a15415"}
Jul 13 18:52:19.851532 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:19.851489    2569 scope.go:117] "RemoveContainer" containerID="1f99bf2136fc57e74e231a5ba5a0423b67ec05c34283fcc048306503b5a15415"
Jul 13 18:52:20.856794 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:20.856752    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-tbqhd" event={"ID":"8977c40b-f5c4-4aea-a441-6611214b30f7","Type":"ContainerStarted","Data":"aa500e1c435167bfe0411cea741a7d44b44c8305426d27fcd19c041e4c407d67"}
Jul 13 18:52:21.018367 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:21.018336    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-586b57644b-q24lm_4fcfa1f0-d0ea-488a-a414-d09b8085aae5/router/0.log"
Jul 13 18:52:21.037313 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:21.037268    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9qzhl_502189f5-f517-4aa2-b359-07b8c5be3b9a/serve-healthcheck-canary/0.log"
Jul 13 18:52:27.716262 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:27.716223    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64b6bd4cb6-m2sqj" podUID="093c2706-82fd-454e-a557-0605e45034db" containerName="console" containerID="cri-o://faa039e77b0cadfb66926d0ba316f152b63a3f98048a1b7794ba6aac5a2c7fdf" gracePeriod=15
Jul 13 18:52:27.719519 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:27.719493    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:52:27.725587 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:27.725555    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-ffb8ff984-rft4x"
Jul 13 18:52:27.886015 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:27.885986    2569 generic.go:356] "Generic (PLEG): container finished" podID="093c2706-82fd-454e-a557-0605e45034db" containerID="faa039e77b0cadfb66926d0ba316f152b63a3f98048a1b7794ba6aac5a2c7fdf" exitCode=0
Jul 13 18:52:27.886189 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:27.886064    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64b6bd4cb6-m2sqj" event={"ID":"093c2706-82fd-454e-a557-0605e45034db","Type":"ContainerDied","Data":"faa039e77b0cadfb66926d0ba316f152b63a3f98048a1b7794ba6aac5a2c7fdf"}
Jul 13 18:52:28.007975 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.007948    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:52:28.120263 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.120229    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-oauth-serving-cert\") pod \"093c2706-82fd-454e-a557-0605e45034db\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") "
Jul 13 18:52:28.120476 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.120273    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/093c2706-82fd-454e-a557-0605e45034db-console-oauth-config\") pod \"093c2706-82fd-454e-a557-0605e45034db\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") "
Jul 13 18:52:28.120476 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.120329    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-console-config\") pod \"093c2706-82fd-454e-a557-0605e45034db\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") "
Jul 13 18:52:28.120476 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.120363    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/093c2706-82fd-454e-a557-0605e45034db-console-serving-cert\") pod \"093c2706-82fd-454e-a557-0605e45034db\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") "
Jul 13 18:52:28.120476 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.120418    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86gnc\" (UniqueName: \"kubernetes.io/projected/093c2706-82fd-454e-a557-0605e45034db-kube-api-access-86gnc\") pod \"093c2706-82fd-454e-a557-0605e45034db\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") "
Jul 13 18:52:28.120476 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.120448    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-trusted-ca-bundle\") pod \"093c2706-82fd-454e-a557-0605e45034db\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") "
Jul 13 18:52:28.120476 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.120475    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-service-ca\") pod \"093c2706-82fd-454e-a557-0605e45034db\" (UID: \"093c2706-82fd-454e-a557-0605e45034db\") "
Jul 13 18:52:28.120791 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.120708    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "093c2706-82fd-454e-a557-0605e45034db" (UID: "093c2706-82fd-454e-a557-0605e45034db"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:52:28.120847 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.120792    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-console-config" (OuterVolumeSpecName: "console-config") pod "093c2706-82fd-454e-a557-0605e45034db" (UID: "093c2706-82fd-454e-a557-0605e45034db"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:52:28.120900 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.120844    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-oauth-serving-cert\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:52:28.121191 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.121147    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "093c2706-82fd-454e-a557-0605e45034db" (UID: "093c2706-82fd-454e-a557-0605e45034db"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:52:28.121349 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.121200    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-service-ca" (OuterVolumeSpecName: "service-ca") pod "093c2706-82fd-454e-a557-0605e45034db" (UID: "093c2706-82fd-454e-a557-0605e45034db"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:52:28.122805 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.122780    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093c2706-82fd-454e-a557-0605e45034db-kube-api-access-86gnc" (OuterVolumeSpecName: "kube-api-access-86gnc") pod "093c2706-82fd-454e-a557-0605e45034db" (UID: "093c2706-82fd-454e-a557-0605e45034db"). InnerVolumeSpecName "kube-api-access-86gnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:52:28.122880 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.122840    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/093c2706-82fd-454e-a557-0605e45034db-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "093c2706-82fd-454e-a557-0605e45034db" (UID: "093c2706-82fd-454e-a557-0605e45034db"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:52:28.122880 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.122860    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/093c2706-82fd-454e-a557-0605e45034db-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "093c2706-82fd-454e-a557-0605e45034db" (UID: "093c2706-82fd-454e-a557-0605e45034db"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:52:28.221489 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.221451    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/093c2706-82fd-454e-a557-0605e45034db-console-oauth-config\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:52:28.221489 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.221485    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-console-config\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:52:28.221489 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.221498    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/093c2706-82fd-454e-a557-0605e45034db-console-serving-cert\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:52:28.221725 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.221511    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86gnc\" (UniqueName: \"kubernetes.io/projected/093c2706-82fd-454e-a557-0605e45034db-kube-api-access-86gnc\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:52:28.221725 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.221524    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-trusted-ca-bundle\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:52:28.221725 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.221536    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/093c2706-82fd-454e-a557-0605e45034db-service-ca\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:52:28.890925 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.890897    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64b6bd4cb6-m2sqj"
Jul 13 18:52:28.890925 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.890913    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64b6bd4cb6-m2sqj" event={"ID":"093c2706-82fd-454e-a557-0605e45034db","Type":"ContainerDied","Data":"a91b8cad75e4c226042e81f796e2cb99b7481d01e98375f1c0f7faa6e56a26f0"}
Jul 13 18:52:28.891448 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.890970    2569 scope.go:117] "RemoveContainer" containerID="faa039e77b0cadfb66926d0ba316f152b63a3f98048a1b7794ba6aac5a2c7fdf"
Jul 13 18:52:28.912532 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.912503    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64b6bd4cb6-m2sqj"]
Jul 13 18:52:28.916507 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:28.916486    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64b6bd4cb6-m2sqj"]
Jul 13 18:52:29.059837 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:29.059802    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="093c2706-82fd-454e-a557-0605e45034db" path="/var/lib/kubelet/pods/093c2706-82fd-454e-a557-0605e45034db/volumes"
Jul 13 18:52:49.604186 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:49.604150    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:49.620464 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:49.620438    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:49.996524 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:52:49.996440    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:07.704616 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.704583    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 18:53:07.705033 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.704924    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="prometheus" containerID="cri-o://6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd" gracePeriod=600
Jul 13 18:53:07.705033 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.704933    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="kube-rbac-proxy" containerID="cri-o://0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76" gracePeriod=600
Jul 13 18:53:07.705033 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.704982    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="kube-rbac-proxy-thanos" containerID="cri-o://16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50" gracePeriod=600
Jul 13 18:53:07.705188 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.704924    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="thanos-sidecar" containerID="cri-o://3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c" gracePeriod=600
Jul 13 18:53:07.705188 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.704935    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="config-reloader" containerID="cri-o://abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc" gracePeriod=600
Jul 13 18:53:07.705188 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.704939    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="kube-rbac-proxy-web" containerID="cri-o://d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96" gracePeriod=600
Jul 13 18:53:07.943899 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.943873    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:07.964651 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.964575    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") "
Jul 13 18:53:07.964651 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.964617    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-config\") pod \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") "
Jul 13 18:53:07.964651 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.964642    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-rulefiles-2\") pod \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") "
Jul 13 18:53:07.964915 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.964666    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-thanos-prometheus-http-client-file\") pod \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") "
Jul 13 18:53:07.964915 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.964812    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-web-config\") pod \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") "
Jul 13 18:53:07.964915 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.964872    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-configmap-kubelet-serving-ca-bundle\") pod \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") "
Jul 13 18:53:07.964915 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.964892    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-trusted-ca-bundle\") pod \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") "
Jul 13 18:53:07.965108 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.964919    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-configmap-serving-certs-ca-bundle\") pod \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") "
Jul 13 18:53:07.965108 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.964952    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-config-out\") pod \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") "
Jul 13 18:53:07.965108 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.964983    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htdn5\" (UniqueName: \"kubernetes.io/projected/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-kube-api-access-htdn5\") pod \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") "
Jul 13 18:53:07.965108 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.965022    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-tls-assets\") pod \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") "
Jul 13 18:53:07.965108 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.965052    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-rulefiles-0\") pod \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") "
Jul 13 18:53:07.965108 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.965049    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" (UID: "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:53:07.965108 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.965085    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") "
Jul 13 18:53:07.965622 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.965115    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-configmap-metrics-client-ca\") pod \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") "
Jul 13 18:53:07.965622 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.965141    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-grpc-tls\") pod \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") "
Jul 13 18:53:07.965622 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.965174    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-prometheus-k8s-tls\") pod \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") "
Jul 13 18:53:07.965622 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.965207    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-metrics-client-certs\") pod \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") "
Jul 13 18:53:07.965622 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.965221    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" (UID: "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:53:07.965622 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.965245    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-db\") pod \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") "
Jul 13 18:53:07.965622 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.965268    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-kube-rbac-proxy\") pod \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") "
Jul 13 18:53:07.965622 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.965334    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-rulefiles-1\") pod \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\" (UID: \"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8\") "
Jul 13 18:53:07.965622 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.965602    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:53:07.965622 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.965624    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:53:07.967394 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.966239    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" (UID: "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:53:07.968106 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.968077    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" (UID: "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:53:07.968211 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.968155    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" (UID: "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:53:07.968211 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.968165    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" (UID: "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:53:07.968606 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.968576    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" (UID: "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:53:07.968694 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.968661    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" (UID: "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:53:07.969070 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.969048    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" (UID: "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:53:07.969589 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.969562    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" (UID: "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:53:07.970422 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.970381    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-config-out" (OuterVolumeSpecName: "config-out") pod "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" (UID: "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:53:07.970422 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.970405    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" (UID: "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:53:07.970560 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.970487    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" (UID: "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:53:07.970681 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.970661    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" (UID: "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:53:07.971144 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.971119    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" (UID: "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:53:07.971144 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.971132    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" (UID: "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:53:07.971418 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.971382    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-kube-api-access-htdn5" (OuterVolumeSpecName: "kube-api-access-htdn5") pod "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" (UID: "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8"). InnerVolumeSpecName "kube-api-access-htdn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:53:07.971504 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.971450    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-config" (OuterVolumeSpecName: "config") pod "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" (UID: "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:53:07.971693 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.971668    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" (UID: "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:53:07.981500 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:07.981472    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-web-config" (OuterVolumeSpecName: "web-config") pod "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" (UID: "5fe8c10d-cc0c-4133-bffb-973b4a57a6c8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:53:08.048898 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.048867    2569 generic.go:356] "Generic (PLEG): container finished" podID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerID="16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50" exitCode=0
Jul 13 18:53:08.048898 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.048891    2569 generic.go:356] "Generic (PLEG): container finished" podID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerID="0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76" exitCode=0
Jul 13 18:53:08.048898 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.048897    2569 generic.go:356] "Generic (PLEG): container finished" podID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerID="d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96" exitCode=0
Jul 13 18:53:08.048898 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.048902    2569 generic.go:356] "Generic (PLEG): container finished" podID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerID="3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c" exitCode=0
Jul 13 18:53:08.048898 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.048907    2569 generic.go:356] "Generic (PLEG): container finished" podID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerID="abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc" exitCode=0
Jul 13 18:53:08.049171 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.048912    2569 generic.go:356] "Generic (PLEG): container finished" podID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerID="6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd" exitCode=0
Jul 13 18:53:08.049171 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.048947    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.049171 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.048956    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8","Type":"ContainerDied","Data":"16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50"}
Jul 13 18:53:08.049171 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.049056    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8","Type":"ContainerDied","Data":"0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76"}
Jul 13 18:53:08.049171 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.049100    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8","Type":"ContainerDied","Data":"d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96"}
Jul 13 18:53:08.049171 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.049129    2569 scope.go:117] "RemoveContainer" containerID="16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50"
Jul 13 18:53:08.049171 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.049141    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8","Type":"ContainerDied","Data":"3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c"}
Jul 13 18:53:08.049403 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.049181    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8","Type":"ContainerDied","Data":"abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc"}
Jul 13 18:53:08.049403 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.049222    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8","Type":"ContainerDied","Data":"6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd"}
Jul 13 18:53:08.049403 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.049262    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5fe8c10d-cc0c-4133-bffb-973b4a57a6c8","Type":"ContainerDied","Data":"d00709f722b69671b4e3f06969f3928c86909d6a5ff181c489bc165124699455"}
Jul 13 18:53:08.057933 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.057913    2569 scope.go:117] "RemoveContainer" containerID="0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76"
Jul 13 18:53:08.065052 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.065033    2569 scope.go:117] "RemoveContainer" containerID="d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96"
Jul 13 18:53:08.066120 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.066096    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:53:08.066227 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.066124    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-configmap-metrics-client-ca\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:53:08.066227 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.066137    2569 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-grpc-tls\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:53:08.066227 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.066150    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:53:08.066227 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.066165    2569 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-metrics-client-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:53:08.066227 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.066178    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-db\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:53:08.066227 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.066190    2569 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-kube-rbac-proxy\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:53:08.066227 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.066198    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:53:08.066227 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.066206    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:53:08.066227 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.066214    2569 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-config\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:53:08.066227 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.066223    2569 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:53:08.066227 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.066231    2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-web-config\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:53:08.066746 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.066238    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:53:08.066746 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.066246    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:53:08.066746 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.066254    2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-config-out\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:53:08.066746 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.066262    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htdn5\" (UniqueName: \"kubernetes.io/projected/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-kube-api-access-htdn5\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:53:08.066746 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.066269    2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-tls-assets\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:53:08.066746 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.066277    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 18:53:08.072446 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.072322    2569 scope.go:117] "RemoveContainer" containerID="3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c"
Jul 13 18:53:08.073788 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.073749    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 18:53:08.079034 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.079014    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 18:53:08.079792 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.079776    2569 scope.go:117] "RemoveContainer" containerID="abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc"
Jul 13 18:53:08.086593 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.086580    2569 scope.go:117] "RemoveContainer" containerID="6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd"
Jul 13 18:53:08.093766 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.093749    2569 scope.go:117] "RemoveContainer" containerID="2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366"
Jul 13 18:53:08.100940 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.100922    2569 scope.go:117] "RemoveContainer" containerID="16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50"
Jul 13 18:53:08.101239 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:53:08.101220    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50\": container with ID starting with 16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50 not found: ID does not exist" containerID="16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50"
Jul 13 18:53:08.101301 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.101249    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50"} err="failed to get container status \"16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50\": rpc error: code = NotFound desc = could not find container \"16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50\": container with ID starting with 16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50 not found: ID does not exist"
Jul 13 18:53:08.101301 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.101269    2569 scope.go:117] "RemoveContainer" containerID="0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76"
Jul 13 18:53:08.101558 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:53:08.101540    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76\": container with ID starting with 0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76 not found: ID does not exist" containerID="0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76"
Jul 13 18:53:08.101613 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.101568    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76"} err="failed to get container status \"0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76\": rpc error: code = NotFound desc = could not find container \"0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76\": container with ID starting with 0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76 not found: ID does not exist"
Jul 13 18:53:08.101613 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.101584    2569 scope.go:117] "RemoveContainer" containerID="d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96"
Jul 13 18:53:08.101821 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:53:08.101805    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96\": container with ID starting with d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96 not found: ID does not exist" containerID="d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96"
Jul 13 18:53:08.101859 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.101825    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96"} err="failed to get container status \"d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96\": rpc error: code = NotFound desc = could not find container \"d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96\": container with ID starting with d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96 not found: ID does not exist"
Jul 13 18:53:08.101859 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.101840    2569 scope.go:117] "RemoveContainer" containerID="3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c"
Jul 13 18:53:08.102054 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:53:08.102039    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c\": container with ID starting with 3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c not found: ID does not exist" containerID="3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c"
Jul 13 18:53:08.102092 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.102058    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c"} err="failed to get container status \"3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c\": rpc error: code = NotFound desc = could not find container \"3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c\": container with ID starting with 3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c not found: ID does not exist"
Jul 13 18:53:08.102092 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.102074    2569 scope.go:117] "RemoveContainer" containerID="abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc"
Jul 13 18:53:08.102315 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:53:08.102278    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc\": container with ID starting with abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc not found: ID does not exist" containerID="abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc"
Jul 13 18:53:08.102367 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.102321    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc"} err="failed to get container status \"abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc\": rpc error: code = NotFound desc = could not find container \"abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc\": container with ID starting with abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc not found: ID does not exist"
Jul 13 18:53:08.102367 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.102340    2569 scope.go:117] "RemoveContainer" containerID="6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd"
Jul 13 18:53:08.102572 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:53:08.102557    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd\": container with ID starting with 6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd not found: ID does not exist" containerID="6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd"
Jul 13 18:53:08.102650 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.102590    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd"} err="failed to get container status \"6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd\": rpc error: code = NotFound desc = could not find container \"6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd\": container with ID starting with 6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd not found: ID does not exist"
Jul 13 18:53:08.102650 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.102603    2569 scope.go:117] "RemoveContainer" containerID="2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366"
Jul 13 18:53:08.102863 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:53:08.102847    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366\": container with ID starting with 2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366 not found: ID does not exist" containerID="2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366"
Jul 13 18:53:08.102904 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.102866    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366"} err="failed to get container status \"2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366\": rpc error: code = NotFound desc = could not find container \"2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366\": container with ID starting with 2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366 not found: ID does not exist"
Jul 13 18:53:08.102904 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.102879    2569 scope.go:117] "RemoveContainer" containerID="16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50"
Jul 13 18:53:08.103139 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.103123    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50"} err="failed to get container status \"16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50\": rpc error: code = NotFound desc = could not find container \"16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50\": container with ID starting with 16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50 not found: ID does not exist"
Jul 13 18:53:08.103182 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.103141    2569 scope.go:117] "RemoveContainer" containerID="0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76"
Jul 13 18:53:08.103365 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.103350    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76"} err="failed to get container status \"0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76\": rpc error: code = NotFound desc = could not find container \"0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76\": container with ID starting with 0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76 not found: ID does not exist"
Jul 13 18:53:08.103434 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.103366    2569 scope.go:117] "RemoveContainer" containerID="d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96"
Jul 13 18:53:08.103602 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.103585    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96"} err="failed to get container status \"d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96\": rpc error: code = NotFound desc = could not find container \"d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96\": container with ID starting with d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96 not found: ID does not exist"
Jul 13 18:53:08.103654 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.103602    2569 scope.go:117] "RemoveContainer" containerID="3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c"
Jul 13 18:53:08.103806 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.103788    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c"} err="failed to get container status \"3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c\": rpc error: code = NotFound desc = could not find container \"3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c\": container with ID starting with 3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c not found: ID does not exist"
Jul 13 18:53:08.103856 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.103806    2569 scope.go:117] "RemoveContainer" containerID="abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc"
Jul 13 18:53:08.104017 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.104001    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc"} err="failed to get container status \"abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc\": rpc error: code = NotFound desc = could not find container \"abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc\": container with ID starting with abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc not found: ID does not exist"
Jul 13 18:53:08.104055 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.104019    2569 scope.go:117] "RemoveContainer" containerID="6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd"
Jul 13 18:53:08.104241 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.104222    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd"} err="failed to get container status \"6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd\": rpc error: code = NotFound desc = could not find container \"6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd\": container with ID starting with 6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd not found: ID does not exist"
Jul 13 18:53:08.104241 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.104240    2569 scope.go:117] "RemoveContainer" containerID="2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366"
Jul 13 18:53:08.104484 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.104467    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366"} err="failed to get container status \"2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366\": rpc error: code = NotFound desc = could not find container \"2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366\": container with ID starting with 2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366 not found: ID does not exist"
Jul 13 18:53:08.104533 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.104484    2569 scope.go:117] "RemoveContainer" containerID="16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50"
Jul 13 18:53:08.104692 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.104671    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50"} err="failed to get container status \"16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50\": rpc error: code = NotFound desc = could not find container \"16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50\": container with ID starting with 16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50 not found: ID does not exist"
Jul 13 18:53:08.104732 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.104692    2569 scope.go:117] "RemoveContainer" containerID="0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76"
Jul 13 18:53:08.104894 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.104877    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76"} err="failed to get container status \"0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76\": rpc error: code = NotFound desc = could not find container \"0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76\": container with ID starting with 0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76 not found: ID does not exist"
Jul 13 18:53:08.104945 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.104894    2569 scope.go:117] "RemoveContainer" containerID="d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96"
Jul 13 18:53:08.105118 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.105098    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96"} err="failed to get container status \"d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96\": rpc error: code = NotFound desc = could not find container \"d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96\": container with ID starting with d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96 not found: ID does not exist"
Jul 13 18:53:08.105153 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.105121    2569 scope.go:117] "RemoveContainer" containerID="3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c"
Jul 13 18:53:08.105318 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.105284    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c"} err="failed to get container status \"3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c\": rpc error: code = NotFound desc = could not find container \"3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c\": container with ID starting with 3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c not found: ID does not exist"
Jul 13 18:53:08.105318 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.105317    2569 scope.go:117] "RemoveContainer" containerID="abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc"
Jul 13 18:53:08.105525 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.105508    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc"} err="failed to get container status \"abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc\": rpc error: code = NotFound desc = could not find container \"abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc\": container with ID starting with abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc not found: ID does not exist"
Jul 13 18:53:08.105525 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.105524    2569 scope.go:117] "RemoveContainer" containerID="6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd"
Jul 13 18:53:08.105705 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.105686    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd"} err="failed to get container status \"6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd\": rpc error: code = NotFound desc = could not find container \"6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd\": container with ID starting with 6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd not found: ID does not exist"
Jul 13 18:53:08.105747 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.105705    2569 scope.go:117] "RemoveContainer" containerID="2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366"
Jul 13 18:53:08.105893 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.105877    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366"} err="failed to get container status \"2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366\": rpc error: code = NotFound desc = could not find container \"2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366\": container with ID starting with 2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366 not found: ID does not exist"
Jul 13 18:53:08.105941 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.105892    2569 scope.go:117] "RemoveContainer" containerID="16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50"
Jul 13 18:53:08.106077 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.106062    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50"} err="failed to get container status \"16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50\": rpc error: code = NotFound desc = could not find container \"16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50\": container with ID starting with 16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50 not found: ID does not exist"
Jul 13 18:53:08.106116 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.106077    2569 scope.go:117] "RemoveContainer" containerID="0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76"
Jul 13 18:53:08.106268 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.106252    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76"} err="failed to get container status \"0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76\": rpc error: code = NotFound desc = could not find container \"0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76\": container with ID starting with 0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76 not found: ID does not exist"
Jul 13 18:53:08.106327 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.106267    2569 scope.go:117] "RemoveContainer" containerID="d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96"
Jul 13 18:53:08.106516 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.106502    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96"} err="failed to get container status \"d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96\": rpc error: code = NotFound desc = could not find container \"d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96\": container with ID starting with d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96 not found: ID does not exist"
Jul 13 18:53:08.106560 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.106516    2569 scope.go:117] "RemoveContainer" containerID="3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c"
Jul 13 18:53:08.106681 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.106666    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c"} err="failed to get container status \"3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c\": rpc error: code = NotFound desc = could not find container \"3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c\": container with ID starting with 3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c not found: ID does not exist"
Jul 13 18:53:08.106727 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.106680    2569 scope.go:117] "RemoveContainer" containerID="abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc"
Jul 13 18:53:08.106900 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.106884    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc"} err="failed to get container status \"abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc\": rpc error: code = NotFound desc = could not find container \"abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc\": container with ID starting with abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc not found: ID does not exist"
Jul 13 18:53:08.107019 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.106900    2569 scope.go:117] "RemoveContainer" containerID="6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd"
Jul 13 18:53:08.107192 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.107169    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd"} err="failed to get container status \"6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd\": rpc error: code = NotFound desc = could not find container \"6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd\": container with ID starting with 6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd not found: ID does not exist"
Jul 13 18:53:08.107443 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.107192    2569 scope.go:117] "RemoveContainer" containerID="2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366"
Jul 13 18:53:08.107592 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.107538    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366"} err="failed to get container status \"2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366\": rpc error: code = NotFound desc = could not find container \"2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366\": container with ID starting with 2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366 not found: ID does not exist"
Jul 13 18:53:08.107592 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.107574    2569 scope.go:117] "RemoveContainer" containerID="16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50"
Jul 13 18:53:08.107842 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.107808    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50"} err="failed to get container status \"16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50\": rpc error: code = NotFound desc = could not find container \"16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50\": container with ID starting with 16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50 not found: ID does not exist"
Jul 13 18:53:08.107842 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.107835    2569 scope.go:117] "RemoveContainer" containerID="0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76"
Jul 13 18:53:08.108090 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.108073    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76"} err="failed to get container status \"0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76\": rpc error: code = NotFound desc = could not find container \"0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76\": container with ID starting with 0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76 not found: ID does not exist"
Jul 13 18:53:08.108090 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.108090    2569 scope.go:117] "RemoveContainer" containerID="d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96"
Jul 13 18:53:08.108353 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.108317    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96"} err="failed to get container status \"d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96\": rpc error: code = NotFound desc = could not find container \"d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96\": container with ID starting with d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96 not found: ID does not exist"
Jul 13 18:53:08.108353 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.108345    2569 scope.go:117] "RemoveContainer" containerID="3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c"
Jul 13 18:53:08.108646 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.108620    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c"} err="failed to get container status \"3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c\": rpc error: code = NotFound desc = could not find container \"3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c\": container with ID starting with 3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c not found: ID does not exist"
Jul 13 18:53:08.108646 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.108641    2569 scope.go:117] "RemoveContainer" containerID="abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc"
Jul 13 18:53:08.108879 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.108863    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc"} err="failed to get container status \"abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc\": rpc error: code = NotFound desc = could not find container \"abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc\": container with ID starting with abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc not found: ID does not exist"
Jul 13 18:53:08.108939 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.108879    2569 scope.go:117] "RemoveContainer" containerID="6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd"
Jul 13 18:53:08.109097 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109080    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd"} err="failed to get container status \"6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd\": rpc error: code = NotFound desc = could not find container \"6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd\": container with ID starting with 6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd not found: ID does not exist"
Jul 13 18:53:08.109138 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109098    2569 scope.go:117] "RemoveContainer" containerID="2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366"
Jul 13 18:53:08.109329 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109312    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 18:53:08.109370 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109347    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366"} err="failed to get container status \"2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366\": rpc error: code = NotFound desc = could not find container \"2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366\": container with ID starting with 2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366 not found: ID does not exist"
Jul 13 18:53:08.109370 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109362    2569 scope.go:117] "RemoveContainer" containerID="16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50"
Jul 13 18:53:08.109582 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109567    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50"} err="failed to get container status \"16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50\": rpc error: code = NotFound desc = could not find container \"16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50\": container with ID starting with 16ef54da747b65ec44fff6a257c4a8d2409ca52f6548c4af72422b878c72ff50 not found: ID does not exist"
Jul 13 18:53:08.109618 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109582    2569 scope.go:117] "RemoveContainer" containerID="0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76"
Jul 13 18:53:08.109618 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109590    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73a82b38-5c25-40f0-9849-4b0be568ae02" containerName="registry"
Jul 13 18:53:08.109618 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109601    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="73a82b38-5c25-40f0-9849-4b0be568ae02" containerName="registry"
Jul 13 18:53:08.109618 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109607    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="kube-rbac-proxy"
Jul 13 18:53:08.109618 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109613    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="kube-rbac-proxy"
Jul 13 18:53:08.109751 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109621    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="init-config-reloader"
Jul 13 18:53:08.109751 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109629    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="init-config-reloader"
Jul 13 18:53:08.109751 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109642    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="kube-rbac-proxy-thanos"
Jul 13 18:53:08.109751 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109651    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="kube-rbac-proxy-thanos"
Jul 13 18:53:08.109751 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109667    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="thanos-sidecar"
Jul 13 18:53:08.109751 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109675    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="thanos-sidecar"
Jul 13 18:53:08.109751 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109685    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="prometheus"
Jul 13 18:53:08.109751 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109692    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="prometheus"
Jul 13 18:53:08.109751 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109699    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="config-reloader"
Jul 13 18:53:08.109751 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109706    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="config-reloader"
Jul 13 18:53:08.109751 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109713    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="kube-rbac-proxy-web"
Jul 13 18:53:08.109751 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109720    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="kube-rbac-proxy-web"
Jul 13 18:53:08.109751 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109737    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="668d7b34-95bb-46ab-a0ba-1d5dd4911ddf" containerName="console"
Jul 13 18:53:08.109751 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109744    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="668d7b34-95bb-46ab-a0ba-1d5dd4911ddf" containerName="console"
Jul 13 18:53:08.109751 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109752    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="093c2706-82fd-454e-a557-0605e45034db" containerName="console"
Jul 13 18:53:08.110382 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109760    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="093c2706-82fd-454e-a557-0605e45034db" containerName="console"
Jul 13 18:53:08.110382 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109827    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="668d7b34-95bb-46ab-a0ba-1d5dd4911ddf" containerName="console"
Jul 13 18:53:08.110382 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109840    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="config-reloader"
Jul 13 18:53:08.110382 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109849    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="prometheus"
Jul 13 18:53:08.110382 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109858    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="kube-rbac-proxy"
Jul 13 18:53:08.110382 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109865    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="kube-rbac-proxy-web"
Jul 13 18:53:08.110382 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109874    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="thanos-sidecar"
Jul 13 18:53:08.110382 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109881    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" containerName="kube-rbac-proxy-thanos"
Jul 13 18:53:08.110382 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109889    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="73a82b38-5c25-40f0-9849-4b0be568ae02" containerName="registry"
Jul 13 18:53:08.110382 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.109897    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="093c2706-82fd-454e-a557-0605e45034db" containerName="console"
Jul 13 18:53:08.112395 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.112254    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76"} err="failed to get container status \"0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76\": rpc error: code = NotFound desc = could not find container \"0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76\": container with ID starting with 0a9cd7cdb69dcad1dc8972d01cffc76410a9fbed9c83d3d95922b1410dbb6f76 not found: ID does not exist"
Jul 13 18:53:08.112395 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.112280    2569 scope.go:117] "RemoveContainer" containerID="d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96"
Jul 13 18:53:08.112933 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.112911    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96"} err="failed to get container status \"d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96\": rpc error: code = NotFound desc = could not find container \"d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96\": container with ID starting with d1be8c88f49769ea0a6246db33c13ac243686840156f6c6b721a2e0fb172db96 not found: ID does not exist"
Jul 13 18:53:08.112933 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.112934    2569 scope.go:117] "RemoveContainer" containerID="3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c"
Jul 13 18:53:08.113193 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.113177    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c"} err="failed to get container status \"3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c\": rpc error: code = NotFound desc = could not find container \"3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c\": container with ID starting with 3023d02f562fd91f77256e5cf533a39e452deb8e806581e883094e785c25169c not found: ID does not exist"
Jul 13 18:53:08.113250 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.113193    2569 scope.go:117] "RemoveContainer" containerID="abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc"
Jul 13 18:53:08.113492 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.113463    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc"} err="failed to get container status \"abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc\": rpc error: code = NotFound desc = could not find container \"abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc\": container with ID starting with abae2ec0ccfe3078afaaaf4f18a9e58ede64a49f7a567a51b3d20450b471ccbc not found: ID does not exist"
Jul 13 18:53:08.113492 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.113487    2569 scope.go:117] "RemoveContainer" containerID="6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd"
Jul 13 18:53:08.113747 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.113729    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd"} err="failed to get container status \"6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd\": rpc error: code = NotFound desc = could not find container \"6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd\": container with ID starting with 6a24fa4b8c403b73c8891526879386598e1414c3f536d8d9d6b8e26197d738dd not found: ID does not exist"
Jul 13 18:53:08.113747 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.113747    2569 scope.go:117] "RemoveContainer" containerID="2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366"
Jul 13 18:53:08.114012 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.113994    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366"} err="failed to get container status \"2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366\": rpc error: code = NotFound desc = could not find container \"2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366\": container with ID starting with 2b7bc6dcc3911e99589d7f061c3d2e16cc9580ca7c7d66e7a8d46884c63b9366 not found: ID does not exist"
Jul 13 18:53:08.115446 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.115427    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.118498 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.118467    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 13 18:53:08.118604 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.118552    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 13 18:53:08.118658 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.118609    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 13 18:53:08.118713 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.118673    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 13 18:53:08.119145 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.119130    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 13 18:53:08.119210 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.119189    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 13 18:53:08.119460 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.119443    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 13 18:53:08.119546 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.119445    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 13 18:53:08.119672 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.119639    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 13 18:53:08.119751 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.119698    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 13 18:53:08.119751 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.119731    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 13 18:53:08.119894 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.119862    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-88659namsui2e\""
Jul 13 18:53:08.120094 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.120081    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pctbk\""
Jul 13 18:53:08.120191 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.120176    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 13 18:53:08.121489 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.121464    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 13 18:53:08.126355 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.126338    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 13 18:53:08.131357 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.131337    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 18:53:08.167331 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.167284    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.167486 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.167345    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.167486 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.167368    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.167486 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.167383    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.167486 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.167400    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.167486 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.167415    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.167486 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.167476    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbcfk\" (UniqueName: \"kubernetes.io/projected/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-kube-api-access-jbcfk\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.167671 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.167510    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.167671 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.167526    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.167671 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.167543    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.167671 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.167563    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.167671 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.167577    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.167671 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.167644    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-web-config\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.167830 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.167704    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-config-out\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.167830 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.167730    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.167830 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.167753    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-config\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.167830 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.167776    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.167830 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.167799    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.167830 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.167816    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.167995 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.167850    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.269267 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.269180    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.269267 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.269228    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbcfk\" (UniqueName: \"kubernetes.io/projected/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-kube-api-access-jbcfk\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.269267 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.269258    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.269550 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.269284    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.269550 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.269336    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.269550 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.269369    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.269550 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.269394    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.269550 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.269419    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-web-config\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.270190 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.270131    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.270314 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.270205    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-config-out\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.270314 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.270250    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.270314 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.270281    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-config\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.270489 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.270330    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.270489 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.270365    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.270489 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.270393    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.270489 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.270429    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.270489 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.270442    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.270489 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.270460    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.270775 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.270517    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.270775 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.270561    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.270775 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.270593    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.270775 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.270626    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.270965 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.270895    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.273479 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.272448    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.273479 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.272507    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-web-config\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.273479 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.273252    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.273479 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.273385    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.273479 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.273397    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.273950 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.273510    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.273950 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.273790    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.273950 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.273851    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.274094 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.274074    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.274193 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.274115    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-config-out\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.274789 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.274747    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.275424 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.275391    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.276091 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.276070    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.276208 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.276191    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.276482 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.276461    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-config\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.276676 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.276653    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.280612 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.280587    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbcfk\" (UniqueName: \"kubernetes.io/projected/a4f3c1eb-933f-44b3-9461-ec3da50e38b6-kube-api-access-jbcfk\") pod \"prometheus-k8s-0\" (UID: \"a4f3c1eb-933f-44b3-9461-ec3da50e38b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.424882 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.424824    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.590449 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:08.589066    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 18:53:08.594339 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:53:08.594307    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f3c1eb_933f_44b3_9461_ec3da50e38b6.slice/crio-4f87566831b0a99af02f0f2d085f1471791483f20805d0f4b3e678dd381ecc0a WatchSource:0}: Error finding container 4f87566831b0a99af02f0f2d085f1471791483f20805d0f4b3e678dd381ecc0a: Status 404 returned error can't find the container with id 4f87566831b0a99af02f0f2d085f1471791483f20805d0f4b3e678dd381ecc0a
Jul 13 18:53:09.055056 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:09.055021    2569 generic.go:356] "Generic (PLEG): container finished" podID="a4f3c1eb-933f-44b3-9461-ec3da50e38b6" containerID="6685288d56b8ef5fc4ade335ee84aa5a27b700b9acdd45b5965c39f36a6f2f9a" exitCode=0
Jul 13 18:53:09.060251 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:09.060182    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fe8c10d-cc0c-4133-bffb-973b4a57a6c8" path="/var/lib/kubelet/pods/5fe8c10d-cc0c-4133-bffb-973b4a57a6c8/volumes"
Jul 13 18:53:09.060910 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:09.060884    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4f3c1eb-933f-44b3-9461-ec3da50e38b6","Type":"ContainerDied","Data":"6685288d56b8ef5fc4ade335ee84aa5a27b700b9acdd45b5965c39f36a6f2f9a"}
Jul 13 18:53:09.061018 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:09.060958    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4f3c1eb-933f-44b3-9461-ec3da50e38b6","Type":"ContainerStarted","Data":"4f87566831b0a99af02f0f2d085f1471791483f20805d0f4b3e678dd381ecc0a"}
Jul 13 18:53:10.061787 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:10.061747    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4f3c1eb-933f-44b3-9461-ec3da50e38b6","Type":"ContainerStarted","Data":"2685e866fd89c0288ae8f1cc8c986bddd14e9ed036943e98b85cb6c929dc33cf"}
Jul 13 18:53:10.062225 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:10.061808    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4f3c1eb-933f-44b3-9461-ec3da50e38b6","Type":"ContainerStarted","Data":"35fcd51aae630188e8081dec1343b75693c3c00758137a70306a005fa2b93dde"}
Jul 13 18:53:10.062225 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:10.061834    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4f3c1eb-933f-44b3-9461-ec3da50e38b6","Type":"ContainerStarted","Data":"0c1e8941a4c9044a46771ecc255b8cc4ee2f8b448d4edee00240daa0010dac43"}
Jul 13 18:53:10.062225 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:10.061858    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4f3c1eb-933f-44b3-9461-ec3da50e38b6","Type":"ContainerStarted","Data":"1d8c7dfabb33109535c560cd059b81e90591cfef028d5804df8681e216fb06a3"}
Jul 13 18:53:10.062225 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:10.061885    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4f3c1eb-933f-44b3-9461-ec3da50e38b6","Type":"ContainerStarted","Data":"614e6c1e41cc98a0a45d1eda7ee1bd405168ef26d0faf8877ffebb996b112794"}
Jul 13 18:53:10.062225 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:10.061908    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4f3c1eb-933f-44b3-9461-ec3da50e38b6","Type":"ContainerStarted","Data":"66872a3e4534d6b7d7156a9260ca9c480ccb8ae45c811c4b089b95cb1907c452"}
Jul 13 18:53:13.425032 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:53:13.424987    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:54:08.425953 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:54:08.425863    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:54:08.441457 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:54:08.441429    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:54:08.473799 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:54:08.473744    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=60.455543553 podStartE2EDuration="1m0.47372346s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.179907ms" isStatefulPod=false podCreationTimestamp="2026-07-13 18:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:53:10.105393677 +0000 UTC m=+211.619192842" watchObservedRunningTime="2026-07-13 18:54:08.47372346 +0000 UTC m=+269.987522626"
Jul 13 18:54:09.289826 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:54:09.289800    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:54:38.912868 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:54:38.912840    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 18:54:38.917368 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:54:38.917341    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 18:54:38.917851 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:54:38.917827    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 18:54:38.922234 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:54:38.922214    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 18:54:38.924788 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:54:38.924763    2569 kubelet.go:1648] "Image garbage collection succeeded"
Jul 13 18:57:44.797917 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:57:44.797879    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-bdf85754c-vt4bs"]
Jul 13 18:57:44.801306 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:57:44.801275    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-bdf85754c-vt4bs"
Jul 13 18:57:44.804050 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:57:44.804031    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 13 18:57:44.804160 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:57:44.804038    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 13 18:57:44.805457 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:57:44.805423    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-rmdwc\""
Jul 13 18:57:44.805457 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:57:44.805430    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 13 18:57:44.810426 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:57:44.810407    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-bdf85754c-vt4bs"]
Jul 13 18:57:44.855064 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:57:44.855029    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz6sb\" (UniqueName: \"kubernetes.io/projected/11d53207-54ca-4ffd-a9ed-623f2e7555ce-kube-api-access-kz6sb\") pod \"llmisvc-controller-manager-bdf85754c-vt4bs\" (UID: \"11d53207-54ca-4ffd-a9ed-623f2e7555ce\") " pod="kserve/llmisvc-controller-manager-bdf85754c-vt4bs"
Jul 13 18:57:44.855221 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:57:44.855100    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11d53207-54ca-4ffd-a9ed-623f2e7555ce-cert\") pod \"llmisvc-controller-manager-bdf85754c-vt4bs\" (UID: \"11d53207-54ca-4ffd-a9ed-623f2e7555ce\") " pod="kserve/llmisvc-controller-manager-bdf85754c-vt4bs"
Jul 13 18:57:44.955509 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:57:44.955476    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11d53207-54ca-4ffd-a9ed-623f2e7555ce-cert\") pod \"llmisvc-controller-manager-bdf85754c-vt4bs\" (UID: \"11d53207-54ca-4ffd-a9ed-623f2e7555ce\") " pod="kserve/llmisvc-controller-manager-bdf85754c-vt4bs"
Jul 13 18:57:44.955693 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:57:44.955526    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz6sb\" (UniqueName: \"kubernetes.io/projected/11d53207-54ca-4ffd-a9ed-623f2e7555ce-kube-api-access-kz6sb\") pod \"llmisvc-controller-manager-bdf85754c-vt4bs\" (UID: \"11d53207-54ca-4ffd-a9ed-623f2e7555ce\") " pod="kserve/llmisvc-controller-manager-bdf85754c-vt4bs"
Jul 13 18:57:44.955693 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:57:44.955637    2569 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 13 18:57:44.955782 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:57:44.955703    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11d53207-54ca-4ffd-a9ed-623f2e7555ce-cert podName:11d53207-54ca-4ffd-a9ed-623f2e7555ce nodeName:}" failed. No retries permitted until 2026-07-13 18:57:45.455686268 +0000 UTC m=+486.969485416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/11d53207-54ca-4ffd-a9ed-623f2e7555ce-cert") pod "llmisvc-controller-manager-bdf85754c-vt4bs" (UID: "11d53207-54ca-4ffd-a9ed-623f2e7555ce") : secret "llmisvc-webhook-server-cert" not found
Jul 13 18:57:44.964148 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:57:44.964126    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz6sb\" (UniqueName: \"kubernetes.io/projected/11d53207-54ca-4ffd-a9ed-623f2e7555ce-kube-api-access-kz6sb\") pod \"llmisvc-controller-manager-bdf85754c-vt4bs\" (UID: \"11d53207-54ca-4ffd-a9ed-623f2e7555ce\") " pod="kserve/llmisvc-controller-manager-bdf85754c-vt4bs"
Jul 13 18:57:45.459118 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:57:45.459081    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11d53207-54ca-4ffd-a9ed-623f2e7555ce-cert\") pod \"llmisvc-controller-manager-bdf85754c-vt4bs\" (UID: \"11d53207-54ca-4ffd-a9ed-623f2e7555ce\") " pod="kserve/llmisvc-controller-manager-bdf85754c-vt4bs"
Jul 13 18:57:45.459325 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:57:45.459201    2569 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 13 18:57:45.459325 ip-10-0-139-134 kubenswrapper[2569]: E0713 18:57:45.459267    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11d53207-54ca-4ffd-a9ed-623f2e7555ce-cert podName:11d53207-54ca-4ffd-a9ed-623f2e7555ce nodeName:}" failed. No retries permitted until 2026-07-13 18:57:46.45925196 +0000 UTC m=+487.973051107 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/11d53207-54ca-4ffd-a9ed-623f2e7555ce-cert") pod "llmisvc-controller-manager-bdf85754c-vt4bs" (UID: "11d53207-54ca-4ffd-a9ed-623f2e7555ce") : secret "llmisvc-webhook-server-cert" not found
Jul 13 18:57:46.468690 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:57:46.468650    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11d53207-54ca-4ffd-a9ed-623f2e7555ce-cert\") pod \"llmisvc-controller-manager-bdf85754c-vt4bs\" (UID: \"11d53207-54ca-4ffd-a9ed-623f2e7555ce\") " pod="kserve/llmisvc-controller-manager-bdf85754c-vt4bs"
Jul 13 18:57:46.471109 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:57:46.471090    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11d53207-54ca-4ffd-a9ed-623f2e7555ce-cert\") pod \"llmisvc-controller-manager-bdf85754c-vt4bs\" (UID: \"11d53207-54ca-4ffd-a9ed-623f2e7555ce\") " pod="kserve/llmisvc-controller-manager-bdf85754c-vt4bs"
Jul 13 18:57:46.612158 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:57:46.612128    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-bdf85754c-vt4bs"
Jul 13 18:57:46.746063 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:57:46.745990    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-bdf85754c-vt4bs"]
Jul 13 18:57:46.748887 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:57:46.748855    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod11d53207_54ca_4ffd_a9ed_623f2e7555ce.slice/crio-a168669d61492004ff7a5a198d75aea8484ebafd748536dc8a48923c06f6eb42 WatchSource:0}: Error finding container a168669d61492004ff7a5a198d75aea8484ebafd748536dc8a48923c06f6eb42: Status 404 returned error can't find the container with id a168669d61492004ff7a5a198d75aea8484ebafd748536dc8a48923c06f6eb42
Jul 13 18:57:46.750221 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:57:46.750205    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 18:57:47.099626 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:57:47.099589    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-bdf85754c-vt4bs" event={"ID":"11d53207-54ca-4ffd-a9ed-623f2e7555ce","Type":"ContainerStarted","Data":"a168669d61492004ff7a5a198d75aea8484ebafd748536dc8a48923c06f6eb42"}
Jul 13 18:57:50.112111 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:57:50.112019    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-bdf85754c-vt4bs" event={"ID":"11d53207-54ca-4ffd-a9ed-623f2e7555ce","Type":"ContainerStarted","Data":"439e163cd1e95fcb3042d88de5ee5ffc876843e47eaf3eb9f79d782d970c77e2"}
Jul 13 18:57:50.112494 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:57:50.112143    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-bdf85754c-vt4bs"
Jul 13 18:57:50.156867 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:57:50.156817    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-bdf85754c-vt4bs" podStartSLOduration=3.06343375 podStartE2EDuration="6.156799242s" totalImagesPullingTime="3.093365492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:57:50.153710028 +0000 UTC m=+491.667509196" watchObservedRunningTime="2026-07-13 18:57:50.156799242 +0000 UTC m=+491.670598404"
Jul 13 18:58:21.119615 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:21.119576    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-bdf85754c-vt4bs"
Jul 13 18:58:42.265710 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.265622    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ghrqc"]
Jul 13 18:58:42.268927 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.268911    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ghrqc"
Jul 13 18:58:42.271596 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.271574    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 13 18:58:42.272826 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.272810    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 13 18:58:42.273026 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.272832    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-bw974\""
Jul 13 18:58:42.278204 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.278171    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ghrqc"]
Jul 13 18:58:42.346390 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.346348    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ffp6\" (UniqueName: \"kubernetes.io/projected/612cb93c-72d4-490d-b125-bdbec7308ecc-kube-api-access-7ffp6\") pod \"cert-manager-cainjector-559cdd8c87-ghrqc\" (UID: \"612cb93c-72d4-490d-b125-bdbec7308ecc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ghrqc"
Jul 13 18:58:42.346572 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.346418    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/612cb93c-72d4-490d-b125-bdbec7308ecc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ghrqc\" (UID: \"612cb93c-72d4-490d-b125-bdbec7308ecc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ghrqc"
Jul 13 18:58:42.361573 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.361537    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vhz7z"]
Jul 13 18:58:42.364747 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.364724    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-vhz7z"
Jul 13 18:58:42.367374 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.367355    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-pcp7m\""
Jul 13 18:58:42.374616 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.374595    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vhz7z"]
Jul 13 18:58:42.447640 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.447603    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/abc26c96-e028-4608-89ab-8f5593bf8f46-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vhz7z\" (UID: \"abc26c96-e028-4608-89ab-8f5593bf8f46\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vhz7z"
Jul 13 18:58:42.447834 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.447672    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ffp6\" (UniqueName: \"kubernetes.io/projected/612cb93c-72d4-490d-b125-bdbec7308ecc-kube-api-access-7ffp6\") pod \"cert-manager-cainjector-559cdd8c87-ghrqc\" (UID: \"612cb93c-72d4-490d-b125-bdbec7308ecc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ghrqc"
Jul 13 18:58:42.447834 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.447735    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/612cb93c-72d4-490d-b125-bdbec7308ecc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ghrqc\" (UID: \"612cb93c-72d4-490d-b125-bdbec7308ecc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ghrqc"
Jul 13 18:58:42.447834 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.447758    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm5cl\" (UniqueName: \"kubernetes.io/projected/abc26c96-e028-4608-89ab-8f5593bf8f46-kube-api-access-jm5cl\") pod \"cert-manager-webhook-6486ffd997-vhz7z\" (UID: \"abc26c96-e028-4608-89ab-8f5593bf8f46\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vhz7z"
Jul 13 18:58:42.457185 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.457162    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/612cb93c-72d4-490d-b125-bdbec7308ecc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ghrqc\" (UID: \"612cb93c-72d4-490d-b125-bdbec7308ecc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ghrqc"
Jul 13 18:58:42.457261 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.457241    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ffp6\" (UniqueName: \"kubernetes.io/projected/612cb93c-72d4-490d-b125-bdbec7308ecc-kube-api-access-7ffp6\") pod \"cert-manager-cainjector-559cdd8c87-ghrqc\" (UID: \"612cb93c-72d4-490d-b125-bdbec7308ecc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ghrqc"
Jul 13 18:58:42.548192 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.548103    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm5cl\" (UniqueName: \"kubernetes.io/projected/abc26c96-e028-4608-89ab-8f5593bf8f46-kube-api-access-jm5cl\") pod \"cert-manager-webhook-6486ffd997-vhz7z\" (UID: \"abc26c96-e028-4608-89ab-8f5593bf8f46\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vhz7z"
Jul 13 18:58:42.548192 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.548149    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/abc26c96-e028-4608-89ab-8f5593bf8f46-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vhz7z\" (UID: \"abc26c96-e028-4608-89ab-8f5593bf8f46\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vhz7z"
Jul 13 18:58:42.557638 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.557609    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/abc26c96-e028-4608-89ab-8f5593bf8f46-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vhz7z\" (UID: \"abc26c96-e028-4608-89ab-8f5593bf8f46\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vhz7z"
Jul 13 18:58:42.557773 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.557683    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm5cl\" (UniqueName: \"kubernetes.io/projected/abc26c96-e028-4608-89ab-8f5593bf8f46-kube-api-access-jm5cl\") pod \"cert-manager-webhook-6486ffd997-vhz7z\" (UID: \"abc26c96-e028-4608-89ab-8f5593bf8f46\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vhz7z"
Jul 13 18:58:42.588512 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.588484    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ghrqc"
Jul 13 18:58:42.673656 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.673622    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-vhz7z"
Jul 13 18:58:42.726516 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.726316    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ghrqc"]
Jul 13 18:58:42.815215 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:42.815153    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vhz7z"]
Jul 13 18:58:42.817778 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:58:42.817737    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabc26c96_e028_4608_89ab_8f5593bf8f46.slice/crio-f155c88386cbc53e452172a12208d9ff68fa519ad2f9db6232485e6a177564de WatchSource:0}: Error finding container f155c88386cbc53e452172a12208d9ff68fa519ad2f9db6232485e6a177564de: Status 404 returned error can't find the container with id f155c88386cbc53e452172a12208d9ff68fa519ad2f9db6232485e6a177564de
Jul 13 18:58:43.317897 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:43.317863    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ghrqc" event={"ID":"612cb93c-72d4-490d-b125-bdbec7308ecc","Type":"ContainerStarted","Data":"bc42aa50492fcbc0a04033c12b5438d4d718e62f7a24ad9d0188933cebe26930"}
Jul 13 18:58:43.318969 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:43.318946    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-vhz7z" event={"ID":"abc26c96-e028-4608-89ab-8f5593bf8f46","Type":"ContainerStarted","Data":"f155c88386cbc53e452172a12208d9ff68fa519ad2f9db6232485e6a177564de"}
Jul 13 18:58:46.335138 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:46.335093    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ghrqc" event={"ID":"612cb93c-72d4-490d-b125-bdbec7308ecc","Type":"ContainerStarted","Data":"67947bc84b922bface7ce6a2349551878b2f9f6e134b5116db015a5965190443"}
Jul 13 18:58:46.336686 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:46.336660    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-vhz7z" event={"ID":"abc26c96-e028-4608-89ab-8f5593bf8f46","Type":"ContainerStarted","Data":"76bb035394e29fd5bad05574a6a184c2b0e3d12c0dbeb783f9bb579111dac136"}
Jul 13 18:58:46.336812 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:46.336781    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-vhz7z"
Jul 13 18:58:46.352763 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:46.352710    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ghrqc" podStartSLOduration=0.918909252 podStartE2EDuration="4.352693192s" totalImagesPullingTime="3.43378394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:58:46.351667219 +0000 UTC m=+547.865466388" watchObservedRunningTime="2026-07-13 18:58:46.352693192 +0000 UTC m=+547.866492359"
Jul 13 18:58:46.369993 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:46.369724    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-vhz7z" podStartSLOduration=1.034038316 podStartE2EDuration="4.369698278s" totalImagesPullingTime="3.335659962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:58:46.368738612 +0000 UTC m=+547.882537778" watchObservedRunningTime="2026-07-13 18:58:46.369698278 +0000 UTC m=+547.883497446"
Jul 13 18:58:52.342915 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:52.342885    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-vhz7z"
Jul 13 18:58:58.078879 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:58.078844    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mrl5g"]
Jul 13 18:58:58.085476 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:58.085452    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mrl5g"
Jul 13 18:58:58.088166 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:58.088142    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 13 18:58:58.088276 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:58.088243    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-k849v\""
Jul 13 18:58:58.089507 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:58.089483    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 13 18:58:58.091406 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:58.091364    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mrl5g"]
Jul 13 18:58:58.183524 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:58.183487    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjpxw\" (UniqueName: \"kubernetes.io/projected/ebbf3625-7f4e-477c-9502-6bb3e5bb11d1-kube-api-access-sjpxw\") pod \"openshift-lws-operator-fd8ccff4c-mrl5g\" (UID: \"ebbf3625-7f4e-477c-9502-6bb3e5bb11d1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mrl5g"
Jul 13 18:58:58.183698 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:58.183561    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebbf3625-7f4e-477c-9502-6bb3e5bb11d1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mrl5g\" (UID: \"ebbf3625-7f4e-477c-9502-6bb3e5bb11d1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mrl5g"
Jul 13 18:58:58.288311 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:58.285339    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjpxw\" (UniqueName: \"kubernetes.io/projected/ebbf3625-7f4e-477c-9502-6bb3e5bb11d1-kube-api-access-sjpxw\") pod \"openshift-lws-operator-fd8ccff4c-mrl5g\" (UID: \"ebbf3625-7f4e-477c-9502-6bb3e5bb11d1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mrl5g"
Jul 13 18:58:58.288311 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:58.285434    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebbf3625-7f4e-477c-9502-6bb3e5bb11d1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mrl5g\" (UID: \"ebbf3625-7f4e-477c-9502-6bb3e5bb11d1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mrl5g"
Jul 13 18:58:58.288311 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:58.285955    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebbf3625-7f4e-477c-9502-6bb3e5bb11d1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mrl5g\" (UID: \"ebbf3625-7f4e-477c-9502-6bb3e5bb11d1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mrl5g"
Jul 13 18:58:58.294770 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:58.294742    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjpxw\" (UniqueName: \"kubernetes.io/projected/ebbf3625-7f4e-477c-9502-6bb3e5bb11d1-kube-api-access-sjpxw\") pod \"openshift-lws-operator-fd8ccff4c-mrl5g\" (UID: \"ebbf3625-7f4e-477c-9502-6bb3e5bb11d1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mrl5g"
Jul 13 18:58:58.394720 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:58.394690    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mrl5g"
Jul 13 18:58:58.539367 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:58.539336    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mrl5g"]
Jul 13 18:58:58.540909 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:58:58.540882    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebbf3625_7f4e_477c_9502_6bb3e5bb11d1.slice/crio-fe0829c94793d8123851971cf05e1fbaf0950134cc5a772b31e24866a6e2d801 WatchSource:0}: Error finding container fe0829c94793d8123851971cf05e1fbaf0950134cc5a772b31e24866a6e2d801: Status 404 returned error can't find the container with id fe0829c94793d8123851971cf05e1fbaf0950134cc5a772b31e24866a6e2d801
Jul 13 18:58:59.389355 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:58:59.389312    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mrl5g" event={"ID":"ebbf3625-7f4e-477c-9502-6bb3e5bb11d1","Type":"ContainerStarted","Data":"fe0829c94793d8123851971cf05e1fbaf0950134cc5a772b31e24866a6e2d801"}
Jul 13 18:59:01.400182 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:01.400139    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mrl5g" event={"ID":"ebbf3625-7f4e-477c-9502-6bb3e5bb11d1","Type":"ContainerStarted","Data":"646836a04f41a713260ead0d096c204367d08f4ac5d376db4d896eea48d86935"}
Jul 13 18:59:01.420190 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:01.420135    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mrl5g" podStartSLOduration=1.025722862 podStartE2EDuration="3.420116234s" totalImagesPullingTime="2.394393372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:59:01.419089325 +0000 UTC m=+562.932888491" watchObservedRunningTime="2026-07-13 18:59:01.420116234 +0000 UTC m=+562.933915400"
Jul 13 18:59:26.183864 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:26.183834    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5ff584b858-jl8gt"]
Jul 13 18:59:26.188942 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:26.188923    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5ff584b858-jl8gt"
Jul 13 18:59:26.200739 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:26.200713    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 13 18:59:26.200870 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:26.200744    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-r6lvn\""
Jul 13 18:59:26.200870 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:26.200813    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 13 18:59:26.200870 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:26.200820    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 13 18:59:26.219321 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:26.219281    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5ff584b858-jl8gt"]
Jul 13 18:59:26.339641 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:26.339603    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f345f616-e936-4b68-9497-0fa2dc3855c9-manager-config\") pod \"lws-controller-manager-5ff584b858-jl8gt\" (UID: \"f345f616-e936-4b68-9497-0fa2dc3855c9\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-jl8gt"
Jul 13 18:59:26.339842 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:26.339666    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z26vf\" (UniqueName: \"kubernetes.io/projected/f345f616-e936-4b68-9497-0fa2dc3855c9-kube-api-access-z26vf\") pod \"lws-controller-manager-5ff584b858-jl8gt\" (UID: \"f345f616-e936-4b68-9497-0fa2dc3855c9\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-jl8gt"
Jul 13 18:59:26.339842 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:26.339695    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f345f616-e936-4b68-9497-0fa2dc3855c9-cert\") pod \"lws-controller-manager-5ff584b858-jl8gt\" (UID: \"f345f616-e936-4b68-9497-0fa2dc3855c9\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-jl8gt"
Jul 13 18:59:26.339842 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:26.339781    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f345f616-e936-4b68-9497-0fa2dc3855c9-metrics-cert\") pod \"lws-controller-manager-5ff584b858-jl8gt\" (UID: \"f345f616-e936-4b68-9497-0fa2dc3855c9\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-jl8gt"
Jul 13 18:59:26.440870 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:26.440773    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z26vf\" (UniqueName: \"kubernetes.io/projected/f345f616-e936-4b68-9497-0fa2dc3855c9-kube-api-access-z26vf\") pod \"lws-controller-manager-5ff584b858-jl8gt\" (UID: \"f345f616-e936-4b68-9497-0fa2dc3855c9\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-jl8gt"
Jul 13 18:59:26.440870 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:26.440818    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f345f616-e936-4b68-9497-0fa2dc3855c9-cert\") pod \"lws-controller-manager-5ff584b858-jl8gt\" (UID: \"f345f616-e936-4b68-9497-0fa2dc3855c9\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-jl8gt"
Jul 13 18:59:26.441099 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:26.440943    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f345f616-e936-4b68-9497-0fa2dc3855c9-metrics-cert\") pod \"lws-controller-manager-5ff584b858-jl8gt\" (UID: \"f345f616-e936-4b68-9497-0fa2dc3855c9\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-jl8gt"
Jul 13 18:59:26.441099 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:26.441066    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f345f616-e936-4b68-9497-0fa2dc3855c9-manager-config\") pod \"lws-controller-manager-5ff584b858-jl8gt\" (UID: \"f345f616-e936-4b68-9497-0fa2dc3855c9\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-jl8gt"
Jul 13 18:59:26.441667 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:26.441645    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f345f616-e936-4b68-9497-0fa2dc3855c9-manager-config\") pod \"lws-controller-manager-5ff584b858-jl8gt\" (UID: \"f345f616-e936-4b68-9497-0fa2dc3855c9\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-jl8gt"
Jul 13 18:59:26.443520 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:26.443496    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f345f616-e936-4b68-9497-0fa2dc3855c9-metrics-cert\") pod \"lws-controller-manager-5ff584b858-jl8gt\" (UID: \"f345f616-e936-4b68-9497-0fa2dc3855c9\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-jl8gt"
Jul 13 18:59:26.443582 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:26.443543    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f345f616-e936-4b68-9497-0fa2dc3855c9-cert\") pod \"lws-controller-manager-5ff584b858-jl8gt\" (UID: \"f345f616-e936-4b68-9497-0fa2dc3855c9\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-jl8gt"
Jul 13 18:59:26.448735 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:26.448714    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z26vf\" (UniqueName: \"kubernetes.io/projected/f345f616-e936-4b68-9497-0fa2dc3855c9-kube-api-access-z26vf\") pod \"lws-controller-manager-5ff584b858-jl8gt\" (UID: \"f345f616-e936-4b68-9497-0fa2dc3855c9\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-jl8gt"
Jul 13 18:59:26.497621 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:26.497581    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5ff584b858-jl8gt"
Jul 13 18:59:26.640423 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:26.640394    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5ff584b858-jl8gt"]
Jul 13 18:59:26.643169 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:59:26.643139    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf345f616_e936_4b68_9497_0fa2dc3855c9.slice/crio-7a7355a3e8770e94b0ce4370e53c4260e207631536f56bb9b7a7f19a7bc217be WatchSource:0}: Error finding container 7a7355a3e8770e94b0ce4370e53c4260e207631536f56bb9b7a7f19a7bc217be: Status 404 returned error can't find the container with id 7a7355a3e8770e94b0ce4370e53c4260e207631536f56bb9b7a7f19a7bc217be
Jul 13 18:59:27.514179 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:27.513960    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5ff584b858-jl8gt" event={"ID":"f345f616-e936-4b68-9497-0fa2dc3855c9","Type":"ContainerStarted","Data":"7a7355a3e8770e94b0ce4370e53c4260e207631536f56bb9b7a7f19a7bc217be"}
Jul 13 18:59:28.521333 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:28.521272    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5ff584b858-jl8gt" event={"ID":"f345f616-e936-4b68-9497-0fa2dc3855c9","Type":"ContainerStarted","Data":"07c6b1ad563f5c1f32d2d3cbd000cf0485ccc0e69f2098aeaf2f536e0d494080"}
Jul 13 18:59:28.521805 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:28.521451    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5ff584b858-jl8gt"
Jul 13 18:59:28.539936 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:28.539882    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5ff584b858-jl8gt" podStartSLOduration=0.892032149 podStartE2EDuration="2.539865532s" totalImagesPullingTime="1.647833383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:59:28.538129792 +0000 UTC m=+590.051928958" watchObservedRunningTime="2026-07-13 18:59:28.539865532 +0000 UTC m=+590.053664766"
Jul 13 18:59:38.941356 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:38.941329    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 18:59:38.942471 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:38.942445    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 18:59:38.945539 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:38.945518    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 18:59:38.946724 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:38.946704    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 18:59:39.527609 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:39.527575    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5ff584b858-jl8gt"
Jul 13 18:59:50.392571 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:50.392534    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6d85f6564-xdn96"]
Jul 13 18:59:50.398922 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:50.398902    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6d85f6564-xdn96"
Jul 13 18:59:50.402997 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:50.402977    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 13 18:59:50.403246 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:50.403229    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 13 18:59:50.403363 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:50.403344    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-6mvkd\""
Jul 13 18:59:50.408721 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:50.408697    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6d85f6564-xdn96"]
Jul 13 18:59:50.434108 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:50.434070    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjgcm\" (UniqueName: \"kubernetes.io/projected/eeaf5dd2-2dd3-48fe-9953-305cb53e5440-kube-api-access-sjgcm\") pod \"authorino-operator-6d85f6564-xdn96\" (UID: \"eeaf5dd2-2dd3-48fe-9953-305cb53e5440\") " pod="kuadrant-system/authorino-operator-6d85f6564-xdn96"
Jul 13 18:59:50.534514 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:50.534482    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjgcm\" (UniqueName: \"kubernetes.io/projected/eeaf5dd2-2dd3-48fe-9953-305cb53e5440-kube-api-access-sjgcm\") pod \"authorino-operator-6d85f6564-xdn96\" (UID: \"eeaf5dd2-2dd3-48fe-9953-305cb53e5440\") " pod="kuadrant-system/authorino-operator-6d85f6564-xdn96"
Jul 13 18:59:50.547530 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:50.547506    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjgcm\" (UniqueName: \"kubernetes.io/projected/eeaf5dd2-2dd3-48fe-9953-305cb53e5440-kube-api-access-sjgcm\") pod \"authorino-operator-6d85f6564-xdn96\" (UID: \"eeaf5dd2-2dd3-48fe-9953-305cb53e5440\") " pod="kuadrant-system/authorino-operator-6d85f6564-xdn96"
Jul 13 18:59:50.709217 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:50.709122    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6d85f6564-xdn96"
Jul 13 18:59:50.884849 ip-10-0-139-134 kubenswrapper[2569]: W0713 18:59:50.884816    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeaf5dd2_2dd3_48fe_9953_305cb53e5440.slice/crio-0bf3c017e80d78ce3f913eac712a2ea22c532c0735db42bfc980d9da4316caa8 WatchSource:0}: Error finding container 0bf3c017e80d78ce3f913eac712a2ea22c532c0735db42bfc980d9da4316caa8: Status 404 returned error can't find the container with id 0bf3c017e80d78ce3f913eac712a2ea22c532c0735db42bfc980d9da4316caa8
Jul 13 18:59:50.890077 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:50.890053    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6d85f6564-xdn96"]
Jul 13 18:59:51.618533 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:51.618500    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6d85f6564-xdn96" event={"ID":"eeaf5dd2-2dd3-48fe-9953-305cb53e5440","Type":"ContainerStarted","Data":"0bf3c017e80d78ce3f913eac712a2ea22c532c0735db42bfc980d9da4316caa8"}
Jul 13 18:59:53.636352 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:53.636314    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6d85f6564-xdn96" event={"ID":"eeaf5dd2-2dd3-48fe-9953-305cb53e5440","Type":"ContainerStarted","Data":"648d875596bef80ffc711a6a2446fc768fe7cce7a2927508abed25b016af506c"}
Jul 13 18:59:53.636798 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:53.636418    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6d85f6564-xdn96"
Jul 13 18:59:53.658961 ip-10-0-139-134 kubenswrapper[2569]: I0713 18:59:53.658888    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6d85f6564-xdn96" podStartSLOduration=1.06448808 podStartE2EDuration="3.658871751s" totalImagesPullingTime="2.594383671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:59:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:59:53.657945503 +0000 UTC m=+615.171744673" watchObservedRunningTime="2026-07-13 18:59:53.658871751 +0000 UTC m=+615.172670915"
Jul 13 19:00:04.643510 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:04.643414    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6d85f6564-xdn96"
Jul 13 19:00:39.210514 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:39.210479    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b6c89bb85-knflr"]
Jul 13 19:00:39.214186 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:39.214168    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-knflr"
Jul 13 19:00:39.217008 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:39.216981    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dfbsq\""
Jul 13 19:00:39.222166 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:39.222136    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-knflr"]
Jul 13 19:00:39.259653 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:39.259618    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8fnh\" (UniqueName: \"kubernetes.io/projected/04d21c25-b549-4687-b934-fd5e4a78992c-kube-api-access-p8fnh\") pod \"authorino-b6c89bb85-knflr\" (UID: \"04d21c25-b549-4687-b934-fd5e4a78992c\") " pod="kuadrant-system/authorino-b6c89bb85-knflr"
Jul 13 19:00:39.360347 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:39.360302    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8fnh\" (UniqueName: \"kubernetes.io/projected/04d21c25-b549-4687-b934-fd5e4a78992c-kube-api-access-p8fnh\") pod \"authorino-b6c89bb85-knflr\" (UID: \"04d21c25-b549-4687-b934-fd5e4a78992c\") " pod="kuadrant-system/authorino-b6c89bb85-knflr"
Jul 13 19:00:39.369549 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:39.369516    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8fnh\" (UniqueName: \"kubernetes.io/projected/04d21c25-b549-4687-b934-fd5e4a78992c-kube-api-access-p8fnh\") pod \"authorino-b6c89bb85-knflr\" (UID: \"04d21c25-b549-4687-b934-fd5e4a78992c\") " pod="kuadrant-system/authorino-b6c89bb85-knflr"
Jul 13 19:00:39.453679 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:39.453644    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8df697c9f-brb5n"]
Jul 13 19:00:39.457110 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:39.457092    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-brb5n"
Jul 13 19:00:39.469990 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:39.469958    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-brb5n"]
Jul 13 19:00:39.523735 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:39.523698    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-knflr"
Jul 13 19:00:39.562896 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:39.562861    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxdcl\" (UniqueName: \"kubernetes.io/projected/6dcce36b-7bc5-4ac1-a217-be602314dda2-kube-api-access-gxdcl\") pod \"authorino-8df697c9f-brb5n\" (UID: \"6dcce36b-7bc5-4ac1-a217-be602314dda2\") " pod="kuadrant-system/authorino-8df697c9f-brb5n"
Jul 13 19:00:39.658816 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:39.658791    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-knflr"]
Jul 13 19:00:39.661252 ip-10-0-139-134 kubenswrapper[2569]: W0713 19:00:39.661226    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04d21c25_b549_4687_b934_fd5e4a78992c.slice/crio-1775fc69ff5621cc3e76cd43ea7d5ce5b8d38e8b32a0040462af81cdbee9c8e5 WatchSource:0}: Error finding container 1775fc69ff5621cc3e76cd43ea7d5ce5b8d38e8b32a0040462af81cdbee9c8e5: Status 404 returned error can't find the container with id 1775fc69ff5621cc3e76cd43ea7d5ce5b8d38e8b32a0040462af81cdbee9c8e5
Jul 13 19:00:39.664355 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:39.664279    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxdcl\" (UniqueName: \"kubernetes.io/projected/6dcce36b-7bc5-4ac1-a217-be602314dda2-kube-api-access-gxdcl\") pod \"authorino-8df697c9f-brb5n\" (UID: \"6dcce36b-7bc5-4ac1-a217-be602314dda2\") " pod="kuadrant-system/authorino-8df697c9f-brb5n"
Jul 13 19:00:39.675573 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:39.675542    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxdcl\" (UniqueName: \"kubernetes.io/projected/6dcce36b-7bc5-4ac1-a217-be602314dda2-kube-api-access-gxdcl\") pod \"authorino-8df697c9f-brb5n\" (UID: \"6dcce36b-7bc5-4ac1-a217-be602314dda2\") " pod="kuadrant-system/authorino-8df697c9f-brb5n"
Jul 13 19:00:39.766385 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:39.766305    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-brb5n"
Jul 13 19:00:39.848185 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:39.848150    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-knflr" event={"ID":"04d21c25-b549-4687-b934-fd5e4a78992c","Type":"ContainerStarted","Data":"1775fc69ff5621cc3e76cd43ea7d5ce5b8d38e8b32a0040462af81cdbee9c8e5"}
Jul 13 19:00:39.907230 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:39.907200    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-brb5n"]
Jul 13 19:00:39.909335 ip-10-0-139-134 kubenswrapper[2569]: W0713 19:00:39.909312    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6dcce36b_7bc5_4ac1_a217_be602314dda2.slice/crio-51f9dffa0efdec7d851313b52875a0a84e831be919a8350f704aff34f38a3da4 WatchSource:0}: Error finding container 51f9dffa0efdec7d851313b52875a0a84e831be919a8350f704aff34f38a3da4: Status 404 returned error can't find the container with id 51f9dffa0efdec7d851313b52875a0a84e831be919a8350f704aff34f38a3da4
Jul 13 19:00:40.857383 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:40.857342    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-brb5n" event={"ID":"6dcce36b-7bc5-4ac1-a217-be602314dda2","Type":"ContainerStarted","Data":"51f9dffa0efdec7d851313b52875a0a84e831be919a8350f704aff34f38a3da4"}
Jul 13 19:00:42.869169 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:42.869118    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-knflr" event={"ID":"04d21c25-b549-4687-b934-fd5e4a78992c","Type":"ContainerStarted","Data":"eb1faf9900158c1fd602ea987e94fbdd31ba28b737f3623a202d63302a35082e"}
Jul 13 19:00:42.870606 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:42.870578    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-brb5n" event={"ID":"6dcce36b-7bc5-4ac1-a217-be602314dda2","Type":"ContainerStarted","Data":"a37981b2e3ac7e38992188ebd03d7112f3727e1d8540d7d9994a0922eaaaa54d"}
Jul 13 19:00:42.886244 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:42.886179    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b6c89bb85-knflr" podStartSLOduration=1.066131255 podStartE2EDuration="3.886161813s" totalImagesPullingTime="2.820030558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:00:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:00:42.884627982 +0000 UTC m=+664.398427147" watchObservedRunningTime="2026-07-13 19:00:42.886161813 +0000 UTC m=+664.399960979"
Jul 13 19:00:42.901661 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:42.901473    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8df697c9f-brb5n" podStartSLOduration=1.340835202 podStartE2EDuration="3.901451502s" totalImagesPullingTime="2.5606163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:00:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:00:42.901039514 +0000 UTC m=+664.414838680" watchObservedRunningTime="2026-07-13 19:00:42.901451502 +0000 UTC m=+664.415250670"
Jul 13 19:00:42.926174 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:42.926140    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-knflr"]
Jul 13 19:00:44.885081 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:44.885033    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b6c89bb85-knflr" podUID="04d21c25-b549-4687-b934-fd5e4a78992c" containerName="authorino" containerID="cri-o://eb1faf9900158c1fd602ea987e94fbdd31ba28b737f3623a202d63302a35082e" gracePeriod=30
Jul 13 19:00:45.138365 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:45.138306    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-knflr"
Jul 13 19:00:45.222712 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:45.222672    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8fnh\" (UniqueName: \"kubernetes.io/projected/04d21c25-b549-4687-b934-fd5e4a78992c-kube-api-access-p8fnh\") pod \"04d21c25-b549-4687-b934-fd5e4a78992c\" (UID: \"04d21c25-b549-4687-b934-fd5e4a78992c\") "
Jul 13 19:00:45.225015 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:45.224984    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d21c25-b549-4687-b934-fd5e4a78992c-kube-api-access-p8fnh" (OuterVolumeSpecName: "kube-api-access-p8fnh") pod "04d21c25-b549-4687-b934-fd5e4a78992c" (UID: "04d21c25-b549-4687-b934-fd5e4a78992c"). InnerVolumeSpecName "kube-api-access-p8fnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:00:45.323234 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:45.323194    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8fnh\" (UniqueName: \"kubernetes.io/projected/04d21c25-b549-4687-b934-fd5e4a78992c-kube-api-access-p8fnh\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:00:45.890794 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:45.890758    2569 generic.go:356] "Generic (PLEG): container finished" podID="04d21c25-b549-4687-b934-fd5e4a78992c" containerID="eb1faf9900158c1fd602ea987e94fbdd31ba28b737f3623a202d63302a35082e" exitCode=0
Jul 13 19:00:45.891190 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:45.890809    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-knflr"
Jul 13 19:00:45.891190 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:45.890839    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-knflr" event={"ID":"04d21c25-b549-4687-b934-fd5e4a78992c","Type":"ContainerDied","Data":"eb1faf9900158c1fd602ea987e94fbdd31ba28b737f3623a202d63302a35082e"}
Jul 13 19:00:45.891190 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:45.890882    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-knflr" event={"ID":"04d21c25-b549-4687-b934-fd5e4a78992c","Type":"ContainerDied","Data":"1775fc69ff5621cc3e76cd43ea7d5ce5b8d38e8b32a0040462af81cdbee9c8e5"}
Jul 13 19:00:45.891190 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:45.890898    2569 scope.go:117] "RemoveContainer" containerID="eb1faf9900158c1fd602ea987e94fbdd31ba28b737f3623a202d63302a35082e"
Jul 13 19:00:45.899993 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:45.899977    2569 scope.go:117] "RemoveContainer" containerID="eb1faf9900158c1fd602ea987e94fbdd31ba28b737f3623a202d63302a35082e"
Jul 13 19:00:45.900260 ip-10-0-139-134 kubenswrapper[2569]: E0713 19:00:45.900242    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb1faf9900158c1fd602ea987e94fbdd31ba28b737f3623a202d63302a35082e\": container with ID starting with eb1faf9900158c1fd602ea987e94fbdd31ba28b737f3623a202d63302a35082e not found: ID does not exist" containerID="eb1faf9900158c1fd602ea987e94fbdd31ba28b737f3623a202d63302a35082e"
Jul 13 19:00:45.900349 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:45.900272    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1faf9900158c1fd602ea987e94fbdd31ba28b737f3623a202d63302a35082e"} err="failed to get container status \"eb1faf9900158c1fd602ea987e94fbdd31ba28b737f3623a202d63302a35082e\": rpc error: code = NotFound desc = could not find container \"eb1faf9900158c1fd602ea987e94fbdd31ba28b737f3623a202d63302a35082e\": container with ID starting with eb1faf9900158c1fd602ea987e94fbdd31ba28b737f3623a202d63302a35082e not found: ID does not exist"
Jul 13 19:00:45.911924 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:45.911899    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-knflr"]
Jul 13 19:00:45.915180 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:45.915160    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-knflr"]
Jul 13 19:00:47.060142 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:00:47.060104    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04d21c25-b549-4687-b934-fd5e4a78992c" path="/var/lib/kubelet/pods/04d21c25-b549-4687-b934-fd5e4a78992c/volumes"
Jul 13 19:02:22.478063 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:22.478025    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c77957575-cfthr"]
Jul 13 19:02:22.478571 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:22.478426    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04d21c25-b549-4687-b934-fd5e4a78992c" containerName="authorino"
Jul 13 19:02:22.478571 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:22.478439    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="04d21c25-b549-4687-b934-fd5e4a78992c" containerName="authorino"
Jul 13 19:02:22.478571 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:22.478496    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="04d21c25-b549-4687-b934-fd5e4a78992c" containerName="authorino"
Jul 13 19:02:22.481517 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:22.481497    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-cfthr"
Jul 13 19:02:22.484260 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:22.484241    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 13 19:02:22.489473 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:22.489444    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-cfthr"]
Jul 13 19:02:22.579341 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:22.579304    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgjbg\" (UniqueName: \"kubernetes.io/projected/0ea71281-4a8f-4c25-b9ef-4daa6f7465ce-kube-api-access-lgjbg\") pod \"authorino-7c77957575-cfthr\" (UID: \"0ea71281-4a8f-4c25-b9ef-4daa6f7465ce\") " pod="kuadrant-system/authorino-7c77957575-cfthr"
Jul 13 19:02:22.579341 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:22.579345    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0ea71281-4a8f-4c25-b9ef-4daa6f7465ce-tls-cert\") pod \"authorino-7c77957575-cfthr\" (UID: \"0ea71281-4a8f-4c25-b9ef-4daa6f7465ce\") " pod="kuadrant-system/authorino-7c77957575-cfthr"
Jul 13 19:02:22.680396 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:22.680359    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgjbg\" (UniqueName: \"kubernetes.io/projected/0ea71281-4a8f-4c25-b9ef-4daa6f7465ce-kube-api-access-lgjbg\") pod \"authorino-7c77957575-cfthr\" (UID: \"0ea71281-4a8f-4c25-b9ef-4daa6f7465ce\") " pod="kuadrant-system/authorino-7c77957575-cfthr"
Jul 13 19:02:22.680396 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:22.680401    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0ea71281-4a8f-4c25-b9ef-4daa6f7465ce-tls-cert\") pod \"authorino-7c77957575-cfthr\" (UID: \"0ea71281-4a8f-4c25-b9ef-4daa6f7465ce\") " pod="kuadrant-system/authorino-7c77957575-cfthr"
Jul 13 19:02:22.683006 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:22.682985    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0ea71281-4a8f-4c25-b9ef-4daa6f7465ce-tls-cert\") pod \"authorino-7c77957575-cfthr\" (UID: \"0ea71281-4a8f-4c25-b9ef-4daa6f7465ce\") " pod="kuadrant-system/authorino-7c77957575-cfthr"
Jul 13 19:02:22.688802 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:22.688775    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgjbg\" (UniqueName: \"kubernetes.io/projected/0ea71281-4a8f-4c25-b9ef-4daa6f7465ce-kube-api-access-lgjbg\") pod \"authorino-7c77957575-cfthr\" (UID: \"0ea71281-4a8f-4c25-b9ef-4daa6f7465ce\") " pod="kuadrant-system/authorino-7c77957575-cfthr"
Jul 13 19:02:22.790896 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:22.790802    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-cfthr"
Jul 13 19:02:22.923774 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:22.923753    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-cfthr"]
Jul 13 19:02:22.926120 ip-10-0-139-134 kubenswrapper[2569]: W0713 19:02:22.926090    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ea71281_4a8f_4c25_b9ef_4daa6f7465ce.slice/crio-15e143b504414b4e155380827761fbbd3ef9d59a7a1ee10a965a80a9278efce6 WatchSource:0}: Error finding container 15e143b504414b4e155380827761fbbd3ef9d59a7a1ee10a965a80a9278efce6: Status 404 returned error can't find the container with id 15e143b504414b4e155380827761fbbd3ef9d59a7a1ee10a965a80a9278efce6
Jul 13 19:02:23.323157 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:23.323119    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-cfthr" event={"ID":"0ea71281-4a8f-4c25-b9ef-4daa6f7465ce","Type":"ContainerStarted","Data":"15e143b504414b4e155380827761fbbd3ef9d59a7a1ee10a965a80a9278efce6"}
Jul 13 19:02:24.335656 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:24.335610    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-cfthr" event={"ID":"0ea71281-4a8f-4c25-b9ef-4daa6f7465ce","Type":"ContainerStarted","Data":"9477a69ccf5338f66b0e9f910a9972fca15834dcf40d35056cce8a6b2b8db59a"}
Jul 13 19:02:24.352161 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:24.352106    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c77957575-cfthr" podStartSLOduration=1.8864457319999999 podStartE2EDuration="2.352091697s" totalImagesPullingTime="465.645965ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:02:24.351335934 +0000 UTC m=+765.865135097" watchObservedRunningTime="2026-07-13 19:02:24.352091697 +0000 UTC m=+765.865890862"
Jul 13 19:02:24.374641 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:24.374603    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-brb5n"]
Jul 13 19:02:24.374867 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:24.374833    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-8df697c9f-brb5n" podUID="6dcce36b-7bc5-4ac1-a217-be602314dda2" containerName="authorino" containerID="cri-o://a37981b2e3ac7e38992188ebd03d7112f3727e1d8540d7d9994a0922eaaaa54d" gracePeriod=30
Jul 13 19:02:24.632516 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:24.632487    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-brb5n"
Jul 13 19:02:24.699167 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:24.699134    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxdcl\" (UniqueName: \"kubernetes.io/projected/6dcce36b-7bc5-4ac1-a217-be602314dda2-kube-api-access-gxdcl\") pod \"6dcce36b-7bc5-4ac1-a217-be602314dda2\" (UID: \"6dcce36b-7bc5-4ac1-a217-be602314dda2\") "
Jul 13 19:02:24.701415 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:24.701379    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dcce36b-7bc5-4ac1-a217-be602314dda2-kube-api-access-gxdcl" (OuterVolumeSpecName: "kube-api-access-gxdcl") pod "6dcce36b-7bc5-4ac1-a217-be602314dda2" (UID: "6dcce36b-7bc5-4ac1-a217-be602314dda2"). InnerVolumeSpecName "kube-api-access-gxdcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:02:24.800137 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:24.800091    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxdcl\" (UniqueName: \"kubernetes.io/projected/6dcce36b-7bc5-4ac1-a217-be602314dda2-kube-api-access-gxdcl\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:02:25.341026 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:25.340991    2569 generic.go:356] "Generic (PLEG): container finished" podID="6dcce36b-7bc5-4ac1-a217-be602314dda2" containerID="a37981b2e3ac7e38992188ebd03d7112f3727e1d8540d7d9994a0922eaaaa54d" exitCode=0
Jul 13 19:02:25.341517 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:25.341033    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-brb5n"
Jul 13 19:02:25.341517 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:25.341081    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-brb5n" event={"ID":"6dcce36b-7bc5-4ac1-a217-be602314dda2","Type":"ContainerDied","Data":"a37981b2e3ac7e38992188ebd03d7112f3727e1d8540d7d9994a0922eaaaa54d"}
Jul 13 19:02:25.341517 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:25.341131    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-brb5n" event={"ID":"6dcce36b-7bc5-4ac1-a217-be602314dda2","Type":"ContainerDied","Data":"51f9dffa0efdec7d851313b52875a0a84e831be919a8350f704aff34f38a3da4"}
Jul 13 19:02:25.341517 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:25.341159    2569 scope.go:117] "RemoveContainer" containerID="a37981b2e3ac7e38992188ebd03d7112f3727e1d8540d7d9994a0922eaaaa54d"
Jul 13 19:02:25.349844 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:25.349815    2569 scope.go:117] "RemoveContainer" containerID="a37981b2e3ac7e38992188ebd03d7112f3727e1d8540d7d9994a0922eaaaa54d"
Jul 13 19:02:25.350148 ip-10-0-139-134 kubenswrapper[2569]: E0713 19:02:25.350125    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a37981b2e3ac7e38992188ebd03d7112f3727e1d8540d7d9994a0922eaaaa54d\": container with ID starting with a37981b2e3ac7e38992188ebd03d7112f3727e1d8540d7d9994a0922eaaaa54d not found: ID does not exist" containerID="a37981b2e3ac7e38992188ebd03d7112f3727e1d8540d7d9994a0922eaaaa54d"
Jul 13 19:02:25.350223 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:25.350157    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a37981b2e3ac7e38992188ebd03d7112f3727e1d8540d7d9994a0922eaaaa54d"} err="failed to get container status \"a37981b2e3ac7e38992188ebd03d7112f3727e1d8540d7d9994a0922eaaaa54d\": rpc error: code = NotFound desc = could not find container \"a37981b2e3ac7e38992188ebd03d7112f3727e1d8540d7d9994a0922eaaaa54d\": container with ID starting with a37981b2e3ac7e38992188ebd03d7112f3727e1d8540d7d9994a0922eaaaa54d not found: ID does not exist"
Jul 13 19:02:25.358405 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:25.358373    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-brb5n"]
Jul 13 19:02:25.362627 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:25.362594    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8df697c9f-brb5n"]
Jul 13 19:02:27.060973 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:02:27.060938    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dcce36b-7bc5-4ac1-a217-be602314dda2" path="/var/lib/kubelet/pods/6dcce36b-7bc5-4ac1-a217-be602314dda2/volumes"
Jul 13 19:04:38.969621 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:04:38.969548    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:04:38.970170 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:04:38.969825    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:04:38.974352 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:04:38.974328    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:04:38.974526 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:04:38.974468    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:05:02.697320 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.697265    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"]
Jul 13 19:05:02.697860 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.697750    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dcce36b-7bc5-4ac1-a217-be602314dda2" containerName="authorino"
Jul 13 19:05:02.697860 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.697764    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dcce36b-7bc5-4ac1-a217-be602314dda2" containerName="authorino"
Jul 13 19:05:02.697987 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.697879    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dcce36b-7bc5-4ac1-a217-be602314dda2" containerName="authorino"
Jul 13 19:05:02.701455 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.701433    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:02.704274 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.704251    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 19:05:02.705713 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.705694    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Jul 13 19:05:02.705836 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.705816    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 19:05:02.705977 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.705963    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gv2sm\""
Jul 13 19:05:02.712464 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.712440    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"]
Jul 13 19:05:02.770450 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.770408    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:02.770673 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.770458    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:02.770673 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.770550    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:02.770673 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.770614    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:02.770673 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.770639    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:02.770673 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.770664    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6bpn\" (UniqueName: \"kubernetes.io/projected/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-kube-api-access-j6bpn\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:02.871833 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.871796    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:02.872056 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.871854    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:02.872056 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.871902    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:02.872056 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.871933    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:02.872056 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.871960    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6bpn\" (UniqueName: \"kubernetes.io/projected/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-kube-api-access-j6bpn\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:02.872056 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.872036    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:02.872328 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.872267    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:02.872433 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.872409    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:02.872496 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.872467    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:02.874436 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.874411    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:02.874685 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.874667    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:02.882430 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:02.882404    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6bpn\" (UniqueName: \"kubernetes.io/projected/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-kube-api-access-j6bpn\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:03.013518 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:03.013420    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:03.158216 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:03.158187    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"]
Jul 13 19:05:03.160409 ip-10-0-139-134 kubenswrapper[2569]: W0713 19:05:03.160373    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d4615cf_ca5a_4ef6_ad7a_bdfb88e8f4c8.slice/crio-009270ad8bde2d52aa04fbe7d483a382509b8f7bb06ce7bdae5ff5908c4aa438 WatchSource:0}: Error finding container 009270ad8bde2d52aa04fbe7d483a382509b8f7bb06ce7bdae5ff5908c4aa438: Status 404 returned error can't find the container with id 009270ad8bde2d52aa04fbe7d483a382509b8f7bb06ce7bdae5ff5908c4aa438
Jul 13 19:05:03.162849 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:03.162832    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 19:05:04.057048 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:04.057013    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6" event={"ID":"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8","Type":"ContainerStarted","Data":"009270ad8bde2d52aa04fbe7d483a382509b8f7bb06ce7bdae5ff5908c4aa438"}
Jul 13 19:05:05.063546 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:05.063507    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6" event={"ID":"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8","Type":"ContainerStarted","Data":"9bd0b4af8e36361667e63cb02439e5cb44862a7dbcc2364d106254bc3ec40113"}
Jul 13 19:05:05.105069 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:05.105014    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6" podStartSLOduration=1.8979835029999998 podStartE2EDuration="3.104995571s" totalImagesPullingTime="1.207012068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:05:05.104984108 +0000 UTC m=+926.618783274" watchObservedRunningTime="2026-07-13 19:05:05.104995571 +0000 UTC m=+926.618794736"
Jul 13 19:05:06.013711 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:06.013671    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:13.014021 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:13.013978    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:13.026779 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:13.026745    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:13.111274 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:13.111244    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:16.610745 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:16.610708    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"]
Jul 13 19:05:22.946652 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:22.946613    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"]
Jul 13 19:05:22.950144 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:22.950125    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:22.953046 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:22.953019    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-zh96m\""
Jul 13 19:05:22.953193 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:22.953059    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 13 19:05:22.963787 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:22.963759    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"]
Jul 13 19:05:23.057953 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.057917    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.058124 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.057990    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.058124 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.058018    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.058124 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.058044    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6s6z\" (UniqueName: \"kubernetes.io/projected/8d041b3e-8b24-4ef1-8550-f814a5d09104-kube-api-access-k6s6z\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.058124 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.058061    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.058275 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.058140    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.058275 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.058165    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.058275 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.058197    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d041b3e-8b24-4ef1-8550-f814a5d09104-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.058275 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.058247    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.159560 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.159522    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.159560 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.159570    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d041b3e-8b24-4ef1-8550-f814a5d09104-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.159803 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.159603    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.159803 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.159639    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.159803 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.159683    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.159803 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.159709    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.159803 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.159742    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6s6z\" (UniqueName: \"kubernetes.io/projected/8d041b3e-8b24-4ef1-8550-f814a5d09104-kube-api-access-k6s6z\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.159803 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.159780    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.160050 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.159847    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.160104 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.160075    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.160158 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.160116    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.160232 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.160200    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.160439 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.160419    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.160563 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.160544    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d041b3e-8b24-4ef1-8550-f814a5d09104-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.162263 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.162239    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.162367 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.162262    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.170639 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.170604    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6s6z\" (UniqueName: \"kubernetes.io/projected/8d041b3e-8b24-4ef1-8550-f814a5d09104-kube-api-access-k6s6z\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.170784 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.170738    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-kgtbh\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.261117 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.261009    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:23.407184 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:23.407148    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"]
Jul 13 19:05:23.410044 ip-10-0-139-134 kubenswrapper[2569]: W0713 19:05:23.410016    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d041b3e_8b24_4ef1_8550_f814a5d09104.slice/crio-ea9e9ecbc1bf24fb73e6293344e81cc075a6c9e423d4f20c5c26df73d80733ee WatchSource:0}: Error finding container ea9e9ecbc1bf24fb73e6293344e81cc075a6c9e423d4f20c5c26df73d80733ee: Status 404 returned error can't find the container with id ea9e9ecbc1bf24fb73e6293344e81cc075a6c9e423d4f20c5c26df73d80733ee
Jul 13 19:05:24.155111 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:24.155072    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh" event={"ID":"8d041b3e-8b24-4ef1-8550-f814a5d09104","Type":"ContainerStarted","Data":"ea9e9ecbc1bf24fb73e6293344e81cc075a6c9e423d4f20c5c26df73d80733ee"}
Jul 13 19:05:25.902774 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:25.902730    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Jul 13 19:05:25.903061 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:25.902814    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Jul 13 19:05:25.903061 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:25.902846    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Jul 13 19:05:26.167178 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:26.167077    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh" event={"ID":"8d041b3e-8b24-4ef1-8550-f814a5d09104","Type":"ContainerStarted","Data":"9c78b563e02a451aeb39bc78ab543daae9bb33fedcb08bb8cb4e4f89285b4148"}
Jul 13 19:05:26.261650 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:26.261608    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:26.263446 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:26.263411    2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh" podUID="8d041b3e-8b24-4ef1-8550-f814a5d09104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 13 19:05:27.266112 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:27.266078    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:27.291498 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:27.291438    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh" podStartSLOduration=2.800936011 podStartE2EDuration="5.291421598s" totalImagesPullingTime="2.490485587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:05:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:05:26.195268543 +0000 UTC m=+947.709067711" watchObservedRunningTime="2026-07-13 19:05:27.291421598 +0000 UTC m=+948.805220765"
Jul 13 19:05:28.176743 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:28.176709    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:28.177683 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:28.177664    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:05:31.633663 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:31.633625    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6" podUID="9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8" containerName="main" containerID="cri-o://9bd0b4af8e36361667e63cb02439e5cb44862a7dbcc2364d106254bc3ec40113" gracePeriod=45
Jul 13 19:05:31.891512 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:31.891447    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:32.047652 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.047612    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-tmp-dir\") pod \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") "
Jul 13 19:05:32.047652 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.047653    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6bpn\" (UniqueName: \"kubernetes.io/projected/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-kube-api-access-j6bpn\") pod \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") "
Jul 13 19:05:32.047908 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.047683    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-tls-certs\") pod \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") "
Jul 13 19:05:32.047908 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.047750    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-home\") pod \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") "
Jul 13 19:05:32.047908 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.047855    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-dshm\") pod \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") "
Jul 13 19:05:32.047908 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.047881    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-model-cache\") pod \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\" (UID: \"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8\") "
Jul 13 19:05:32.047908 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.047877    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8" (UID: "9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:05:32.048140 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.048056    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-home" (OuterVolumeSpecName: "home") pod "9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8" (UID: "9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:05:32.048182 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.048132    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8" (UID: "9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:05:32.048182 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.048141    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-tmp-dir\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:05:32.050150 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.050122    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8" (UID: "9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:05:32.050150 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.050135    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-kube-api-access-j6bpn" (OuterVolumeSpecName: "kube-api-access-j6bpn") pod "9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8" (UID: "9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8"). InnerVolumeSpecName "kube-api-access-j6bpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:05:32.050328 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.050127    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-dshm" (OuterVolumeSpecName: "dshm") pod "9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8" (UID: "9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:05:32.149606 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.149517    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-dshm\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:05:32.149606 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.149546    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-model-cache\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:05:32.149606 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.149559    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6bpn\" (UniqueName: \"kubernetes.io/projected/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-kube-api-access-j6bpn\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:05:32.149606 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.149568    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-tls-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:05:32.149606 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.149577    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8-home\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:05:32.200313 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.200251    2569 generic.go:356] "Generic (PLEG): container finished" podID="9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8" containerID="9bd0b4af8e36361667e63cb02439e5cb44862a7dbcc2364d106254bc3ec40113" exitCode=0
Jul 13 19:05:32.200481 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.200332    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"
Jul 13 19:05:32.200481 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.200337    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6" event={"ID":"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8","Type":"ContainerDied","Data":"9bd0b4af8e36361667e63cb02439e5cb44862a7dbcc2364d106254bc3ec40113"}
Jul 13 19:05:32.200481 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.200420    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6" event={"ID":"9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8","Type":"ContainerDied","Data":"009270ad8bde2d52aa04fbe7d483a382509b8f7bb06ce7bdae5ff5908c4aa438"}
Jul 13 19:05:32.200481 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.200444    2569 scope.go:117] "RemoveContainer" containerID="9bd0b4af8e36361667e63cb02439e5cb44862a7dbcc2364d106254bc3ec40113"
Jul 13 19:05:32.210124 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.210097    2569 scope.go:117] "RemoveContainer" containerID="9bd0b4af8e36361667e63cb02439e5cb44862a7dbcc2364d106254bc3ec40113"
Jul 13 19:05:32.210521 ip-10-0-139-134 kubenswrapper[2569]: E0713 19:05:32.210498    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd0b4af8e36361667e63cb02439e5cb44862a7dbcc2364d106254bc3ec40113\": container with ID starting with 9bd0b4af8e36361667e63cb02439e5cb44862a7dbcc2364d106254bc3ec40113 not found: ID does not exist" containerID="9bd0b4af8e36361667e63cb02439e5cb44862a7dbcc2364d106254bc3ec40113"
Jul 13 19:05:32.210613 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.210530    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd0b4af8e36361667e63cb02439e5cb44862a7dbcc2364d106254bc3ec40113"} err="failed to get container status \"9bd0b4af8e36361667e63cb02439e5cb44862a7dbcc2364d106254bc3ec40113\": rpc error: code = NotFound desc = could not find container \"9bd0b4af8e36361667e63cb02439e5cb44862a7dbcc2364d106254bc3ec40113\": container with ID starting with 9bd0b4af8e36361667e63cb02439e5cb44862a7dbcc2364d106254bc3ec40113 not found: ID does not exist"
Jul 13 19:05:32.224079 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.224043    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"]
Jul 13 19:05:32.232189 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:32.232148    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6"]
Jul 13 19:05:33.061389 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:05:33.061351    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8" path="/var/lib/kubelet/pods/9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8/volumes"
Jul 13 19:06:58.432263 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:06:58.432195    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq"]
Jul 13 19:06:58.432719 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:06:58.432639    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8" containerName="main"
Jul 13 19:06:58.432719 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:06:58.432653    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8" containerName="main"
Jul 13 19:06:58.432719 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:06:58.432712    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d4615cf-ca5a-4ef6-ad7a-bdfb88e8f4c8" containerName="main"
Jul 13 19:06:58.435229 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:06:58.435209    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq"
Jul 13 19:06:58.439316 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:06:58.439271    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 13 19:06:58.439464 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:06:58.439271    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gv2sm\""
Jul 13 19:06:58.439464 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:06:58.439322    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-vctqf\""
Jul 13 19:06:58.448627 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:06:58.448599    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq"]
Jul 13 19:06:58.480420 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:06:58.480384    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32d8eb60-2c82-4477-8b08-48031856f5f5-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq\" (UID: \"32d8eb60-2c82-4477-8b08-48031856f5f5\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq"
Jul 13 19:06:58.480590 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:06:58.480458    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwjs5\" (UniqueName: \"kubernetes.io/projected/32d8eb60-2c82-4477-8b08-48031856f5f5-kube-api-access-vwjs5\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq\" (UID: \"32d8eb60-2c82-4477-8b08-48031856f5f5\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq"
Jul 13 19:06:58.480590 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:06:58.480494    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/32d8eb60-2c82-4477-8b08-48031856f5f5-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq\" (UID: \"32d8eb60-2c82-4477-8b08-48031856f5f5\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq"
Jul 13 19:06:58.581905 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:06:58.581863    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32d8eb60-2c82-4477-8b08-48031856f5f5-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq\" (UID: \"32d8eb60-2c82-4477-8b08-48031856f5f5\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq"
Jul 13 19:06:58.582078 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:06:58.581978    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwjs5\" (UniqueName: \"kubernetes.io/projected/32d8eb60-2c82-4477-8b08-48031856f5f5-kube-api-access-vwjs5\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq\" (UID: \"32d8eb60-2c82-4477-8b08-48031856f5f5\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq"
Jul 13 19:06:58.582078 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:06:58.582016    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/32d8eb60-2c82-4477-8b08-48031856f5f5-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq\" (UID: \"32d8eb60-2c82-4477-8b08-48031856f5f5\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq"
Jul 13 19:06:58.582467 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:06:58.582445    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/32d8eb60-2c82-4477-8b08-48031856f5f5-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq\" (UID: \"32d8eb60-2c82-4477-8b08-48031856f5f5\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq"
Jul 13 19:06:58.584685 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:06:58.584657    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32d8eb60-2c82-4477-8b08-48031856f5f5-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq\" (UID: \"32d8eb60-2c82-4477-8b08-48031856f5f5\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq"
Jul 13 19:06:58.591615 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:06:58.591584    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwjs5\" (UniqueName: \"kubernetes.io/projected/32d8eb60-2c82-4477-8b08-48031856f5f5-kube-api-access-vwjs5\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq\" (UID: \"32d8eb60-2c82-4477-8b08-48031856f5f5\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq"
Jul 13 19:06:58.744955 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:06:58.744868    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq"
Jul 13 19:06:58.883086 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:06:58.883056    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq"]
Jul 13 19:06:58.885105 ip-10-0-139-134 kubenswrapper[2569]: W0713 19:06:58.885076    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d8eb60_2c82_4477_8b08_48031856f5f5.slice/crio-a87976a65fa1683bceb614cb798489dc78c1df2b3410a9f4f2c7c05e720ecee7 WatchSource:0}: Error finding container a87976a65fa1683bceb614cb798489dc78c1df2b3410a9f4f2c7c05e720ecee7: Status 404 returned error can't find the container with id a87976a65fa1683bceb614cb798489dc78c1df2b3410a9f4f2c7c05e720ecee7
Jul 13 19:06:59.609150 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:06:59.609117    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq" event={"ID":"32d8eb60-2c82-4477-8b08-48031856f5f5","Type":"ContainerStarted","Data":"a87976a65fa1683bceb614cb798489dc78c1df2b3410a9f4f2c7c05e720ecee7"}
Jul 13 19:07:00.615602 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:00.615557    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq" event={"ID":"32d8eb60-2c82-4477-8b08-48031856f5f5","Type":"ContainerStarted","Data":"7370fe7300b7328e42c83986fec1af93c4e055a2bcbe2d55ceec18ed64581eac"}
Jul 13 19:07:00.616051 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:00.615665    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq"
Jul 13 19:07:00.634498 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:00.634438    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq" podStartSLOduration=1.428807179 podStartE2EDuration="2.634417524s" totalImagesPullingTime="1.205610345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:06:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:07:00.63281606 +0000 UTC m=+1042.146615219" watchObservedRunningTime="2026-07-13 19:07:00.634417524 +0000 UTC m=+1042.148216690"
Jul 13 19:07:31.621944 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:31.621867    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq"
Jul 13 19:07:33.009603 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:33.009564    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq"]
Jul 13 19:07:33.018767 ip-10-0-139-134 kubenswrapper[2569]: E0713 19:07:33.018714    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 19:07:33.018767 ip-10-0-139-134 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:07:33.018767 ip-10-0-139-134 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 13 19:07:33.018767 ip-10-0-139-134 kubenswrapper[2569]:  > containerID="7370fe7300b7328e42c83986fec1af93c4e055a2bcbe2d55ceec18ed64581eac" cmd=["/bin/sleep","15"]
Jul 13 19:07:33.019018 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:33.018798    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq" message="" err=<
Jul 13 19:07:33.019018 ip-10-0-139-134 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:07:33.019018 ip-10-0-139-134 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 13 19:07:33.019018 ip-10-0-139-134 kubenswrapper[2569]:  >
Jul 13 19:07:33.019018 ip-10-0-139-134 kubenswrapper[2569]: E0713 19:07:33.018815    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 19:07:33.019018 ip-10-0-139-134 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:07:33.019018 ip-10-0-139-134 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 13 19:07:33.019018 ip-10-0-139-134 kubenswrapper[2569]:  > pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq" podUID="32d8eb60-2c82-4477-8b08-48031856f5f5" containerName="main" containerID="cri-o://7370fe7300b7328e42c83986fec1af93c4e055a2bcbe2d55ceec18ed64581eac"
Jul 13 19:07:33.019018 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:33.018874    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq" podUID="32d8eb60-2c82-4477-8b08-48031856f5f5" containerName="main" containerID="cri-o://7370fe7300b7328e42c83986fec1af93c4e055a2bcbe2d55ceec18ed64581eac" gracePeriod=60
Jul 13 19:07:33.276667 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:33.276636    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq"
Jul 13 19:07:33.413030 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:33.412994    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwjs5\" (UniqueName: \"kubernetes.io/projected/32d8eb60-2c82-4477-8b08-48031856f5f5-kube-api-access-vwjs5\") pod \"32d8eb60-2c82-4477-8b08-48031856f5f5\" (UID: \"32d8eb60-2c82-4477-8b08-48031856f5f5\") "
Jul 13 19:07:33.413199 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:33.413044    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32d8eb60-2c82-4477-8b08-48031856f5f5-tls-certs\") pod \"32d8eb60-2c82-4477-8b08-48031856f5f5\" (UID: \"32d8eb60-2c82-4477-8b08-48031856f5f5\") "
Jul 13 19:07:33.413199 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:33.413093    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/32d8eb60-2c82-4477-8b08-48031856f5f5-tokenizer-uds\") pod \"32d8eb60-2c82-4477-8b08-48031856f5f5\" (UID: \"32d8eb60-2c82-4477-8b08-48031856f5f5\") "
Jul 13 19:07:33.413451 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:33.413427    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d8eb60-2c82-4477-8b08-48031856f5f5-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "32d8eb60-2c82-4477-8b08-48031856f5f5" (UID: "32d8eb60-2c82-4477-8b08-48031856f5f5"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:07:33.415372 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:33.415346    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d8eb60-2c82-4477-8b08-48031856f5f5-kube-api-access-vwjs5" (OuterVolumeSpecName: "kube-api-access-vwjs5") pod "32d8eb60-2c82-4477-8b08-48031856f5f5" (UID: "32d8eb60-2c82-4477-8b08-48031856f5f5"). InnerVolumeSpecName "kube-api-access-vwjs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:07:33.415484 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:33.415388    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32d8eb60-2c82-4477-8b08-48031856f5f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32d8eb60-2c82-4477-8b08-48031856f5f5" (UID: "32d8eb60-2c82-4477-8b08-48031856f5f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:07:33.514387 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:33.514269    2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/32d8eb60-2c82-4477-8b08-48031856f5f5-tokenizer-uds\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:07:33.514387 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:33.514329    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwjs5\" (UniqueName: \"kubernetes.io/projected/32d8eb60-2c82-4477-8b08-48031856f5f5-kube-api-access-vwjs5\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:07:33.514387 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:33.514341    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32d8eb60-2c82-4477-8b08-48031856f5f5-tls-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:07:33.767243 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:33.767157    2569 generic.go:356] "Generic (PLEG): container finished" podID="32d8eb60-2c82-4477-8b08-48031856f5f5" containerID="7370fe7300b7328e42c83986fec1af93c4e055a2bcbe2d55ceec18ed64581eac" exitCode=0
Jul 13 19:07:33.767243 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:33.767202    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq"
Jul 13 19:07:33.767243 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:33.767231    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq" event={"ID":"32d8eb60-2c82-4477-8b08-48031856f5f5","Type":"ContainerDied","Data":"7370fe7300b7328e42c83986fec1af93c4e055a2bcbe2d55ceec18ed64581eac"}
Jul 13 19:07:33.767489 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:33.767306    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq" event={"ID":"32d8eb60-2c82-4477-8b08-48031856f5f5","Type":"ContainerDied","Data":"a87976a65fa1683bceb614cb798489dc78c1df2b3410a9f4f2c7c05e720ecee7"}
Jul 13 19:07:33.767489 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:33.767335    2569 scope.go:117] "RemoveContainer" containerID="7370fe7300b7328e42c83986fec1af93c4e055a2bcbe2d55ceec18ed64581eac"
Jul 13 19:07:33.777028 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:33.777008    2569 scope.go:117] "RemoveContainer" containerID="7370fe7300b7328e42c83986fec1af93c4e055a2bcbe2d55ceec18ed64581eac"
Jul 13 19:07:33.777347 ip-10-0-139-134 kubenswrapper[2569]: E0713 19:07:33.777315    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7370fe7300b7328e42c83986fec1af93c4e055a2bcbe2d55ceec18ed64581eac\": container with ID starting with 7370fe7300b7328e42c83986fec1af93c4e055a2bcbe2d55ceec18ed64581eac not found: ID does not exist" containerID="7370fe7300b7328e42c83986fec1af93c4e055a2bcbe2d55ceec18ed64581eac"
Jul 13 19:07:33.777434 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:33.777355    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7370fe7300b7328e42c83986fec1af93c4e055a2bcbe2d55ceec18ed64581eac"} err="failed to get container status \"7370fe7300b7328e42c83986fec1af93c4e055a2bcbe2d55ceec18ed64581eac\": rpc error: code = NotFound desc = could not find container \"7370fe7300b7328e42c83986fec1af93c4e055a2bcbe2d55ceec18ed64581eac\": container with ID starting with 7370fe7300b7328e42c83986fec1af93c4e055a2bcbe2d55ceec18ed64581eac not found: ID does not exist"
Jul 13 19:07:33.792213 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:33.792182    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq"]
Jul 13 19:07:33.797685 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:33.797654    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq"]
Jul 13 19:07:35.060678 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:07:35.060643    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32d8eb60-2c82-4477-8b08-48031856f5f5" path="/var/lib/kubelet/pods/32d8eb60-2c82-4477-8b08-48031856f5f5/volumes"
Jul 13 19:08:02.336754 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.336723    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf"]
Jul 13 19:08:02.337180 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.337093    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32d8eb60-2c82-4477-8b08-48031856f5f5" containerName="main"
Jul 13 19:08:02.337180 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.337105    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="32d8eb60-2c82-4477-8b08-48031856f5f5" containerName="main"
Jul 13 19:08:02.337180 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.337170    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="32d8eb60-2c82-4477-8b08-48031856f5f5" containerName="main"
Jul 13 19:08:02.341849 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.341826    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf"
Jul 13 19:08:02.347025 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.346996    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 13 19:08:02.347025 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.347013    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gv2sm\""
Jul 13 19:08:02.347216 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.347012    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-vvnm5\""
Jul 13 19:08:02.353866 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.353836    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf"]
Jul 13 19:08:02.459558 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.459510    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b0ce1481-f8ef-4b1f-85ad-302004756619-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf\" (UID: \"b0ce1481-f8ef-4b1f-85ad-302004756619\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf"
Jul 13 19:08:02.459761 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.459582    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0ce1481-f8ef-4b1f-85ad-302004756619-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf\" (UID: \"b0ce1481-f8ef-4b1f-85ad-302004756619\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf"
Jul 13 19:08:02.459761 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.459660    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j44l\" (UniqueName: \"kubernetes.io/projected/b0ce1481-f8ef-4b1f-85ad-302004756619-kube-api-access-5j44l\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf\" (UID: \"b0ce1481-f8ef-4b1f-85ad-302004756619\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf"
Jul 13 19:08:02.561054 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.561013    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0ce1481-f8ef-4b1f-85ad-302004756619-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf\" (UID: \"b0ce1481-f8ef-4b1f-85ad-302004756619\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf"
Jul 13 19:08:02.561239 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.561076    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j44l\" (UniqueName: \"kubernetes.io/projected/b0ce1481-f8ef-4b1f-85ad-302004756619-kube-api-access-5j44l\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf\" (UID: \"b0ce1481-f8ef-4b1f-85ad-302004756619\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf"
Jul 13 19:08:02.561239 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.561114    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b0ce1481-f8ef-4b1f-85ad-302004756619-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf\" (UID: \"b0ce1481-f8ef-4b1f-85ad-302004756619\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf"
Jul 13 19:08:02.561535 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.561518    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b0ce1481-f8ef-4b1f-85ad-302004756619-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf\" (UID: \"b0ce1481-f8ef-4b1f-85ad-302004756619\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf"
Jul 13 19:08:02.563838 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.563813    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0ce1481-f8ef-4b1f-85ad-302004756619-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf\" (UID: \"b0ce1481-f8ef-4b1f-85ad-302004756619\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf"
Jul 13 19:08:02.570467 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.570436    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j44l\" (UniqueName: \"kubernetes.io/projected/b0ce1481-f8ef-4b1f-85ad-302004756619-kube-api-access-5j44l\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf\" (UID: \"b0ce1481-f8ef-4b1f-85ad-302004756619\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf"
Jul 13 19:08:02.651599 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.651555    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf"
Jul 13 19:08:02.800371 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.800339    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf"]
Jul 13 19:08:02.801890 ip-10-0-139-134 kubenswrapper[2569]: W0713 19:08:02.801862    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ce1481_f8ef_4b1f_85ad_302004756619.slice/crio-4124d3c7ee3f0b981edf4c9a133891b164a90dde5929ebb08fcb2b316813c402 WatchSource:0}: Error finding container 4124d3c7ee3f0b981edf4c9a133891b164a90dde5929ebb08fcb2b316813c402: Status 404 returned error can't find the container with id 4124d3c7ee3f0b981edf4c9a133891b164a90dde5929ebb08fcb2b316813c402
Jul 13 19:08:02.898985 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.898928    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf" event={"ID":"b0ce1481-f8ef-4b1f-85ad-302004756619","Type":"ContainerStarted","Data":"68bbe814f34e5606711659c591b06ed423c116cfcefc808210553bc199258fd8"}
Jul 13 19:08:02.899231 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.898997    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf" event={"ID":"b0ce1481-f8ef-4b1f-85ad-302004756619","Type":"ContainerStarted","Data":"4124d3c7ee3f0b981edf4c9a133891b164a90dde5929ebb08fcb2b316813c402"}
Jul 13 19:08:02.899231 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.899029    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf"
Jul 13 19:08:02.925624 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:02.925518    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf" podStartSLOduration=0.925501075 podStartE2EDuration="925.501075ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:08:02.924014445 +0000 UTC m=+1104.437813612" watchObservedRunningTime="2026-07-13 19:08:02.925501075 +0000 UTC m=+1104.439300240"
Jul 13 19:08:33.905124 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:33.905095    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf"
Jul 13 19:08:41.606643 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.606608    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"]
Jul 13 19:08:41.610020 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.610003    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:41.613615 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.613591    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 13 19:08:41.619349 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.619321    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"]
Jul 13 19:08:41.694708 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.694675    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c067518-9f3d-4218-890f-780279cead75-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:41.694918 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.694736    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:41.694918 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.694778    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:41.694918 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.694799    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:41.694918 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.694820    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:41.694918 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.694882    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpm44\" (UniqueName: \"kubernetes.io/projected/4c067518-9f3d-4218-890f-780279cead75-kube-api-access-cpm44\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:41.795316 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.795244    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:41.795523 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.795343    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:41.795523 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.795406    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpm44\" (UniqueName: \"kubernetes.io/projected/4c067518-9f3d-4218-890f-780279cead75-kube-api-access-cpm44\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:41.795523 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.795466    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c067518-9f3d-4218-890f-780279cead75-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:41.795697 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.795522    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:41.795697 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.795557    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:41.795800 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.795727    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:41.795851 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.795838    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:41.795908 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.795882    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:41.797836 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.797808    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:41.798202 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.798183    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c067518-9f3d-4218-890f-780279cead75-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:41.805327 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.805279    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpm44\" (UniqueName: \"kubernetes.io/projected/4c067518-9f3d-4218-890f-780279cead75-kube-api-access-cpm44\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:41.921613 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:41.921519    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:42.070961 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:42.070936    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"]
Jul 13 19:08:42.072957 ip-10-0-139-134 kubenswrapper[2569]: W0713 19:08:42.072927    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c067518_9f3d_4218_890f_780279cead75.slice/crio-6be4b07eb85f4a5849f50c9630ff7315446fea5720f91ca3d00b446812c054f3 WatchSource:0}: Error finding container 6be4b07eb85f4a5849f50c9630ff7315446fea5720f91ca3d00b446812c054f3: Status 404 returned error can't find the container with id 6be4b07eb85f4a5849f50c9630ff7315446fea5720f91ca3d00b446812c054f3
Jul 13 19:08:42.082238 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:42.082210    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt" event={"ID":"4c067518-9f3d-4218-890f-780279cead75","Type":"ContainerStarted","Data":"6be4b07eb85f4a5849f50c9630ff7315446fea5720f91ca3d00b446812c054f3"}
Jul 13 19:08:43.089366 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:43.089321    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt" event={"ID":"4c067518-9f3d-4218-890f-780279cead75","Type":"ContainerStarted","Data":"9235cb364732cbe05023cd7b8f56caa632cb2bc257372519b643ed5749afd3ab"}
Jul 13 19:08:43.114758 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:43.114704    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt" podStartSLOduration=2.11468346 podStartE2EDuration="2.11468346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:08:43.11187491 +0000 UTC m=+1144.625674075" watchObservedRunningTime="2026-07-13 19:08:43.11468346 +0000 UTC m=+1144.628482625"
Jul 13 19:08:43.922476 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:43.922426    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:51.922347 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:51.922312    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:51.934847 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:51.934816    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:08:52.141866 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:08:52.141834    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:09:15.309402 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:15.309323    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"]
Jul 13 19:09:30.331676 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:30.331606    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt" podUID="4c067518-9f3d-4218-890f-780279cead75" containerName="main" containerID="cri-o://9235cb364732cbe05023cd7b8f56caa632cb2bc257372519b643ed5749afd3ab" gracePeriod=45
Jul 13 19:09:30.591159 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:30.591134    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:09:30.664069 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:30.664026    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-model-cache\") pod \"4c067518-9f3d-4218-890f-780279cead75\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") "
Jul 13 19:09:30.664273 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:30.664129    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-dshm\") pod \"4c067518-9f3d-4218-890f-780279cead75\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") "
Jul 13 19:09:30.664273 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:30.664188    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-tmp-dir\") pod \"4c067518-9f3d-4218-890f-780279cead75\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") "
Jul 13 19:09:30.664273 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:30.664227    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c067518-9f3d-4218-890f-780279cead75-tls-certs\") pod \"4c067518-9f3d-4218-890f-780279cead75\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") "
Jul 13 19:09:30.664273 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:30.664256    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpm44\" (UniqueName: \"kubernetes.io/projected/4c067518-9f3d-4218-890f-780279cead75-kube-api-access-cpm44\") pod \"4c067518-9f3d-4218-890f-780279cead75\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") "
Jul 13 19:09:30.664534 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:30.664282    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-home\") pod \"4c067518-9f3d-4218-890f-780279cead75\" (UID: \"4c067518-9f3d-4218-890f-780279cead75\") "
Jul 13 19:09:30.664534 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:30.664369    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-model-cache" (OuterVolumeSpecName: "model-cache") pod "4c067518-9f3d-4218-890f-780279cead75" (UID: "4c067518-9f3d-4218-890f-780279cead75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:09:30.664637 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:30.664576    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4c067518-9f3d-4218-890f-780279cead75" (UID: "4c067518-9f3d-4218-890f-780279cead75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:09:30.664637 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:30.664593    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-home" (OuterVolumeSpecName: "home") pod "4c067518-9f3d-4218-890f-780279cead75" (UID: "4c067518-9f3d-4218-890f-780279cead75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:09:30.664637 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:30.664622    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-model-cache\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:09:30.666464 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:30.666433    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-dshm" (OuterVolumeSpecName: "dshm") pod "4c067518-9f3d-4218-890f-780279cead75" (UID: "4c067518-9f3d-4218-890f-780279cead75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:09:30.666581 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:30.666502    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c067518-9f3d-4218-890f-780279cead75-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c067518-9f3d-4218-890f-780279cead75" (UID: "4c067518-9f3d-4218-890f-780279cead75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:09:30.666581 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:30.666524    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c067518-9f3d-4218-890f-780279cead75-kube-api-access-cpm44" (OuterVolumeSpecName: "kube-api-access-cpm44") pod "4c067518-9f3d-4218-890f-780279cead75" (UID: "4c067518-9f3d-4218-890f-780279cead75"). InnerVolumeSpecName "kube-api-access-cpm44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:09:30.765219 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:30.765186    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c067518-9f3d-4218-890f-780279cead75-tls-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:09:30.765219 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:30.765214    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpm44\" (UniqueName: \"kubernetes.io/projected/4c067518-9f3d-4218-890f-780279cead75-kube-api-access-cpm44\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:09:30.765219 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:30.765225    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-home\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:09:30.765472 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:30.765235    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-dshm\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:09:30.765472 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:30.765270    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c067518-9f3d-4218-890f-780279cead75-tmp-dir\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:09:31.318566 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:31.318527    2569 generic.go:356] "Generic (PLEG): container finished" podID="4c067518-9f3d-4218-890f-780279cead75" containerID="9235cb364732cbe05023cd7b8f56caa632cb2bc257372519b643ed5749afd3ab" exitCode=0
Jul 13 19:09:31.318765 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:31.318576    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"
Jul 13 19:09:31.318765 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:31.318618    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt" event={"ID":"4c067518-9f3d-4218-890f-780279cead75","Type":"ContainerDied","Data":"9235cb364732cbe05023cd7b8f56caa632cb2bc257372519b643ed5749afd3ab"}
Jul 13 19:09:31.318765 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:31.318673    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt" event={"ID":"4c067518-9f3d-4218-890f-780279cead75","Type":"ContainerDied","Data":"6be4b07eb85f4a5849f50c9630ff7315446fea5720f91ca3d00b446812c054f3"}
Jul 13 19:09:31.318765 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:31.318694    2569 scope.go:117] "RemoveContainer" containerID="9235cb364732cbe05023cd7b8f56caa632cb2bc257372519b643ed5749afd3ab"
Jul 13 19:09:31.327677 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:31.327658    2569 scope.go:117] "RemoveContainer" containerID="9235cb364732cbe05023cd7b8f56caa632cb2bc257372519b643ed5749afd3ab"
Jul 13 19:09:31.327956 ip-10-0-139-134 kubenswrapper[2569]: E0713 19:09:31.327937    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9235cb364732cbe05023cd7b8f56caa632cb2bc257372519b643ed5749afd3ab\": container with ID starting with 9235cb364732cbe05023cd7b8f56caa632cb2bc257372519b643ed5749afd3ab not found: ID does not exist" containerID="9235cb364732cbe05023cd7b8f56caa632cb2bc257372519b643ed5749afd3ab"
Jul 13 19:09:31.328022 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:31.327966    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9235cb364732cbe05023cd7b8f56caa632cb2bc257372519b643ed5749afd3ab"} err="failed to get container status \"9235cb364732cbe05023cd7b8f56caa632cb2bc257372519b643ed5749afd3ab\": rpc error: code = NotFound desc = could not find container \"9235cb364732cbe05023cd7b8f56caa632cb2bc257372519b643ed5749afd3ab\": container with ID starting with 9235cb364732cbe05023cd7b8f56caa632cb2bc257372519b643ed5749afd3ab not found: ID does not exist"
Jul 13 19:09:31.343786 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:31.343755    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"]
Jul 13 19:09:31.345392 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:31.345372    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt"]
Jul 13 19:09:33.060459 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:33.060415    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c067518-9f3d-4218-890f-780279cead75" path="/var/lib/kubelet/pods/4c067518-9f3d-4218-890f-780279cead75/volumes"
Jul 13 19:09:35.664748 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:35.664713    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf"]
Jul 13 19:09:35.677277 ip-10-0-139-134 kubenswrapper[2569]: E0713 19:09:35.677239    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 19:09:35.677277 ip-10-0-139-134 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:09:35.677277 ip-10-0-139-134 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 13 19:09:35.677277 ip-10-0-139-134 kubenswrapper[2569]:  > containerID="68bbe814f34e5606711659c591b06ed423c116cfcefc808210553bc199258fd8" cmd=["/bin/sleep","15"]
Jul 13 19:09:35.677549 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:35.677456    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf" message="" err=<
Jul 13 19:09:35.677549 ip-10-0-139-134 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:09:35.677549 ip-10-0-139-134 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 13 19:09:35.677549 ip-10-0-139-134 kubenswrapper[2569]:  >
Jul 13 19:09:35.677549 ip-10-0-139-134 kubenswrapper[2569]: E0713 19:09:35.677474    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 19:09:35.677549 ip-10-0-139-134 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:09:35.677549 ip-10-0-139-134 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 13 19:09:35.677549 ip-10-0-139-134 kubenswrapper[2569]:  > pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf" podUID="b0ce1481-f8ef-4b1f-85ad-302004756619" containerName="main" containerID="cri-o://68bbe814f34e5606711659c591b06ed423c116cfcefc808210553bc199258fd8"
Jul 13 19:09:35.677549 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:35.677517    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf" podUID="b0ce1481-f8ef-4b1f-85ad-302004756619" containerName="main" containerID="cri-o://68bbe814f34e5606711659c591b06ed423c116cfcefc808210553bc199258fd8" gracePeriod=60
Jul 13 19:09:35.936624 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:35.936597    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf"
Jul 13 19:09:36.013999 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:36.013962    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5j44l\" (UniqueName: \"kubernetes.io/projected/b0ce1481-f8ef-4b1f-85ad-302004756619-kube-api-access-5j44l\") pod \"b0ce1481-f8ef-4b1f-85ad-302004756619\" (UID: \"b0ce1481-f8ef-4b1f-85ad-302004756619\") "
Jul 13 19:09:36.014196 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:36.014082    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0ce1481-f8ef-4b1f-85ad-302004756619-tls-certs\") pod \"b0ce1481-f8ef-4b1f-85ad-302004756619\" (UID: \"b0ce1481-f8ef-4b1f-85ad-302004756619\") "
Jul 13 19:09:36.014196 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:36.014127    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b0ce1481-f8ef-4b1f-85ad-302004756619-tokenizer-uds\") pod \"b0ce1481-f8ef-4b1f-85ad-302004756619\" (UID: \"b0ce1481-f8ef-4b1f-85ad-302004756619\") "
Jul 13 19:09:36.014503 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:36.014452    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ce1481-f8ef-4b1f-85ad-302004756619-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b0ce1481-f8ef-4b1f-85ad-302004756619" (UID: "b0ce1481-f8ef-4b1f-85ad-302004756619"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:09:36.016211 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:36.016180    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ce1481-f8ef-4b1f-85ad-302004756619-kube-api-access-5j44l" (OuterVolumeSpecName: "kube-api-access-5j44l") pod "b0ce1481-f8ef-4b1f-85ad-302004756619" (UID: "b0ce1481-f8ef-4b1f-85ad-302004756619"). InnerVolumeSpecName "kube-api-access-5j44l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:09:36.016327 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:36.016269    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ce1481-f8ef-4b1f-85ad-302004756619-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0ce1481-f8ef-4b1f-85ad-302004756619" (UID: "b0ce1481-f8ef-4b1f-85ad-302004756619"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:09:36.115761 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:36.115719    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0ce1481-f8ef-4b1f-85ad-302004756619-tls-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:09:36.115761 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:36.115751    2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b0ce1481-f8ef-4b1f-85ad-302004756619-tokenizer-uds\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:09:36.115761 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:36.115760    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j44l\" (UniqueName: \"kubernetes.io/projected/b0ce1481-f8ef-4b1f-85ad-302004756619-kube-api-access-5j44l\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:09:36.347504 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:36.347466    2569 generic.go:356] "Generic (PLEG): container finished" podID="b0ce1481-f8ef-4b1f-85ad-302004756619" containerID="68bbe814f34e5606711659c591b06ed423c116cfcefc808210553bc199258fd8" exitCode=0
Jul 13 19:09:36.347704 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:36.347534    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf"
Jul 13 19:09:36.347704 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:36.347550    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf" event={"ID":"b0ce1481-f8ef-4b1f-85ad-302004756619","Type":"ContainerDied","Data":"68bbe814f34e5606711659c591b06ed423c116cfcefc808210553bc199258fd8"}
Jul 13 19:09:36.347704 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:36.347642    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf" event={"ID":"b0ce1481-f8ef-4b1f-85ad-302004756619","Type":"ContainerDied","Data":"4124d3c7ee3f0b981edf4c9a133891b164a90dde5929ebb08fcb2b316813c402"}
Jul 13 19:09:36.347704 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:36.347669    2569 scope.go:117] "RemoveContainer" containerID="68bbe814f34e5606711659c591b06ed423c116cfcefc808210553bc199258fd8"
Jul 13 19:09:36.357147 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:36.357127    2569 scope.go:117] "RemoveContainer" containerID="68bbe814f34e5606711659c591b06ed423c116cfcefc808210553bc199258fd8"
Jul 13 19:09:36.357482 ip-10-0-139-134 kubenswrapper[2569]: E0713 19:09:36.357462    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68bbe814f34e5606711659c591b06ed423c116cfcefc808210553bc199258fd8\": container with ID starting with 68bbe814f34e5606711659c591b06ed423c116cfcefc808210553bc199258fd8 not found: ID does not exist" containerID="68bbe814f34e5606711659c591b06ed423c116cfcefc808210553bc199258fd8"
Jul 13 19:09:36.357543 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:36.357492    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68bbe814f34e5606711659c591b06ed423c116cfcefc808210553bc199258fd8"} err="failed to get container status \"68bbe814f34e5606711659c591b06ed423c116cfcefc808210553bc199258fd8\": rpc error: code = NotFound desc = could not find container \"68bbe814f34e5606711659c591b06ed423c116cfcefc808210553bc199258fd8\": container with ID starting with 68bbe814f34e5606711659c591b06ed423c116cfcefc808210553bc199258fd8 not found: ID does not exist"
Jul 13 19:09:36.369337 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:36.369306    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf"]
Jul 13 19:09:36.375580 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:36.375549    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf"]
Jul 13 19:09:37.060957 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:37.060920    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0ce1481-f8ef-4b1f-85ad-302004756619" path="/var/lib/kubelet/pods/b0ce1481-f8ef-4b1f-85ad-302004756619/volumes"
Jul 13 19:09:38.998166 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:38.998137    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:09:38.998611 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:38.998138    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:09:39.002875 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:39.002853    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:09:39.003036 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:09:39.002853    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:10:07.712771 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:07.712736    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk"]
Jul 13 19:10:07.713263 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:07.713222    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0ce1481-f8ef-4b1f-85ad-302004756619" containerName="main"
Jul 13 19:10:07.713263 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:07.713240    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0ce1481-f8ef-4b1f-85ad-302004756619" containerName="main"
Jul 13 19:10:07.713378 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:07.713270    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c067518-9f3d-4218-890f-780279cead75" containerName="main"
Jul 13 19:10:07.713378 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:07.713280    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c067518-9f3d-4218-890f-780279cead75" containerName="main"
Jul 13 19:10:07.713497 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:07.713380    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0ce1481-f8ef-4b1f-85ad-302004756619" containerName="main"
Jul 13 19:10:07.713497 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:07.713393    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c067518-9f3d-4218-890f-780279cead75" containerName="main"
Jul 13 19:10:07.716236 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:07.716217    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk"
Jul 13 19:10:07.719070 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:07.719044    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 13 19:10:07.720487 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:07.720464    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-fn2cn\""
Jul 13 19:10:07.720603 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:07.720466    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gv2sm\""
Jul 13 19:10:07.728973 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:07.728939    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk"]
Jul 13 19:10:07.811169 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:07.811127    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6dca69aa-72ad-4a4c-97ff-9b0681efd4ed-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk\" (UID: \"6dca69aa-72ad-4a4c-97ff-9b0681efd4ed\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk"
Jul 13 19:10:07.811407 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:07.811259    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch5rj\" (UniqueName: \"kubernetes.io/projected/6dca69aa-72ad-4a4c-97ff-9b0681efd4ed-kube-api-access-ch5rj\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk\" (UID: \"6dca69aa-72ad-4a4c-97ff-9b0681efd4ed\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk"
Jul 13 19:10:07.811407 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:07.811333    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dca69aa-72ad-4a4c-97ff-9b0681efd4ed-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk\" (UID: \"6dca69aa-72ad-4a4c-97ff-9b0681efd4ed\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk"
Jul 13 19:10:07.912018 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:07.911961    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch5rj\" (UniqueName: \"kubernetes.io/projected/6dca69aa-72ad-4a4c-97ff-9b0681efd4ed-kube-api-access-ch5rj\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk\" (UID: \"6dca69aa-72ad-4a4c-97ff-9b0681efd4ed\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk"
Jul 13 19:10:07.912018 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:07.912018    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dca69aa-72ad-4a4c-97ff-9b0681efd4ed-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk\" (UID: \"6dca69aa-72ad-4a4c-97ff-9b0681efd4ed\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk"
Jul 13 19:10:07.912258 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:07.912054    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6dca69aa-72ad-4a4c-97ff-9b0681efd4ed-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk\" (UID: \"6dca69aa-72ad-4a4c-97ff-9b0681efd4ed\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk"
Jul 13 19:10:07.912463 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:07.912446    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6dca69aa-72ad-4a4c-97ff-9b0681efd4ed-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk\" (UID: \"6dca69aa-72ad-4a4c-97ff-9b0681efd4ed\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk"
Jul 13 19:10:07.914758 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:07.914733    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dca69aa-72ad-4a4c-97ff-9b0681efd4ed-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk\" (UID: \"6dca69aa-72ad-4a4c-97ff-9b0681efd4ed\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk"
Jul 13 19:10:07.921923 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:07.921894    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch5rj\" (UniqueName: \"kubernetes.io/projected/6dca69aa-72ad-4a4c-97ff-9b0681efd4ed-kube-api-access-ch5rj\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk\" (UID: \"6dca69aa-72ad-4a4c-97ff-9b0681efd4ed\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk"
Jul 13 19:10:08.026245 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:08.026138    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk"
Jul 13 19:10:08.180625 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:08.180593    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk"]
Jul 13 19:10:08.183851 ip-10-0-139-134 kubenswrapper[2569]: W0713 19:10:08.183820    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dca69aa_72ad_4a4c_97ff_9b0681efd4ed.slice/crio-9e515f9a1f2b333dcaf3d9e2ea0abeab9b3822e07a03cd8df8c17c8dc8479425 WatchSource:0}: Error finding container 9e515f9a1f2b333dcaf3d9e2ea0abeab9b3822e07a03cd8df8c17c8dc8479425: Status 404 returned error can't find the container with id 9e515f9a1f2b333dcaf3d9e2ea0abeab9b3822e07a03cd8df8c17c8dc8479425
Jul 13 19:10:08.497594 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:08.497562    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk" event={"ID":"6dca69aa-72ad-4a4c-97ff-9b0681efd4ed","Type":"ContainerStarted","Data":"3d76b5f3342f1594c09dec42e63fdafaa047c1d3b71d60955fca00799c99fe71"}
Jul 13 19:10:08.497796 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:08.497611    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk" event={"ID":"6dca69aa-72ad-4a4c-97ff-9b0681efd4ed","Type":"ContainerStarted","Data":"9e515f9a1f2b333dcaf3d9e2ea0abeab9b3822e07a03cd8df8c17c8dc8479425"}
Jul 13 19:10:08.497796 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:08.497662    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk"
Jul 13 19:10:08.518125 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:08.518058    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk" podStartSLOduration=1.5180380169999999 podStartE2EDuration="1.518038017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:10:08.516016643 +0000 UTC m=+1230.029815803" watchObservedRunningTime="2026-07-13 19:10:08.518038017 +0000 UTC m=+1230.031837185"
Jul 13 19:10:39.504696 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:10:39.504664    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk"
Jul 13 19:11:55.123754 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.123715    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"]
Jul 13 19:11:55.127486 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.127462    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:11:55.130798 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.130767    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 13 19:11:55.146382 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.146349    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"]
Jul 13 19:11:55.176930 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.176893    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:11:55.177124 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.176952    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:11:55.177124 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.177013    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl2dr\" (UniqueName: \"kubernetes.io/projected/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-kube-api-access-hl2dr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:11:55.177124 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.177031    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:11:55.177124 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.177054    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:11:55.177124 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.177069    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:11:55.278515 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.278474    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:11:55.278716 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.278541    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl2dr\" (UniqueName: \"kubernetes.io/projected/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-kube-api-access-hl2dr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:11:55.278716 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.278564    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:11:55.278716 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.278590    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:11:55.278716 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.278606    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:11:55.278933 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.278759    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:11:55.279033 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.279011    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:11:55.279102 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.279052    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:11:55.279102 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.279086    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:11:55.281044 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.281022    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:11:55.281279 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.281262    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:11:55.288433 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.288412    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl2dr\" (UniqueName: \"kubernetes.io/projected/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-kube-api-access-hl2dr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:11:55.438694 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.438569    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:11:55.593651 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:55.593617    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"]
Jul 13 19:11:55.597149 ip-10-0-139-134 kubenswrapper[2569]: W0713 19:11:55.597120    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d0d9ac_8cf7_4d65_b4cf_1438a4fdb507.slice/crio-613fa7bbe4daa0ebf8bf15009e2fd3277b0c8ba2cac66869d15aa49e5f54794f WatchSource:0}: Error finding container 613fa7bbe4daa0ebf8bf15009e2fd3277b0c8ba2cac66869d15aa49e5f54794f: Status 404 returned error can't find the container with id 613fa7bbe4daa0ebf8bf15009e2fd3277b0c8ba2cac66869d15aa49e5f54794f
Jul 13 19:11:56.002931 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:56.002893    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x" event={"ID":"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507","Type":"ContainerStarted","Data":"9ffe476b1b398071ab3bb5e6f0c2e6148f9d64ebdc9ea8c4f973719ff21fdfe0"}
Jul 13 19:11:56.003113 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:56.002949    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x" event={"ID":"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507","Type":"ContainerStarted","Data":"613fa7bbe4daa0ebf8bf15009e2fd3277b0c8ba2cac66869d15aa49e5f54794f"}
Jul 13 19:11:56.030431 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:56.030372    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x" podStartSLOduration=1.030353993 podStartE2EDuration="1.030353993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:11:56.028956772 +0000 UTC m=+1337.542755939" watchObservedRunningTime="2026-07-13 19:11:56.030353993 +0000 UTC m=+1337.544153158"
Jul 13 19:11:56.439042 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:11:56.438994    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:12:05.439619 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:05.439535    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:12:05.452467 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:05.452432    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:12:06.063462 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:06.063431    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:12:32.992900 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:32.992849    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"]
Jul 13 19:12:48.015546 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.015496    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x" podUID="76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507" containerName="main" containerID="cri-o://9ffe476b1b398071ab3bb5e6f0c2e6148f9d64ebdc9ea8c4f973719ff21fdfe0" gracePeriod=45
Jul 13 19:12:48.052561 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.052520    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x" podUID="76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 13 19:12:48.254854 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.254825    2569 generic.go:356] "Generic (PLEG): container finished" podID="76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507" containerID="9ffe476b1b398071ab3bb5e6f0c2e6148f9d64ebdc9ea8c4f973719ff21fdfe0" exitCode=0
Jul 13 19:12:48.254971 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.254870    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x" event={"ID":"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507","Type":"ContainerDied","Data":"9ffe476b1b398071ab3bb5e6f0c2e6148f9d64ebdc9ea8c4f973719ff21fdfe0"}
Jul 13 19:12:48.254971 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.254911    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x" event={"ID":"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507","Type":"ContainerDied","Data":"613fa7bbe4daa0ebf8bf15009e2fd3277b0c8ba2cac66869d15aa49e5f54794f"}
Jul 13 19:12:48.254971 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.254929    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="613fa7bbe4daa0ebf8bf15009e2fd3277b0c8ba2cac66869d15aa49e5f54794f"
Jul 13 19:12:48.267794 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.267727    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:12:48.374000 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.373962    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-tmp-dir\") pod \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") "
Jul 13 19:12:48.374198 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.374066    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hl2dr\" (UniqueName: \"kubernetes.io/projected/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-kube-api-access-hl2dr\") pod \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") "
Jul 13 19:12:48.374198 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.374106    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-home\") pod \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") "
Jul 13 19:12:48.374198 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.374124    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-model-cache\") pod \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") "
Jul 13 19:12:48.374198 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.374147    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-dshm\") pod \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") "
Jul 13 19:12:48.374198 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.374163    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-tls-certs\") pod \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\" (UID: \"76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507\") "
Jul 13 19:12:48.374490 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.374263    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507" (UID: "76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:12:48.374490 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.374418    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-home" (OuterVolumeSpecName: "home") pod "76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507" (UID: "76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:12:48.374490 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.374434    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-model-cache" (OuterVolumeSpecName: "model-cache") pod "76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507" (UID: "76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:12:48.374644 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.374510    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-tmp-dir\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:12:48.374644 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.374529    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-home\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:12:48.374644 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.374541    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-model-cache\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:12:48.376408 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.376384    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-dshm" (OuterVolumeSpecName: "dshm") pod "76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507" (UID: "76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:12:48.376497 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.376415    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507" (UID: "76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:12:48.376542 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.376527    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-kube-api-access-hl2dr" (OuterVolumeSpecName: "kube-api-access-hl2dr") pod "76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507" (UID: "76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507"). InnerVolumeSpecName "kube-api-access-hl2dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:12:48.475236 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.475191    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl2dr\" (UniqueName: \"kubernetes.io/projected/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-kube-api-access-hl2dr\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:12:48.475236 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.475224    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-dshm\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:12:48.475236 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:48.475237    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507-tls-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:12:49.259469 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:49.259440    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"
Jul 13 19:12:49.287364 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:49.287329    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"]
Jul 13 19:12:49.295587 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:49.295559    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x"]
Jul 13 19:12:51.061578 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:51.061544    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507" path="/var/lib/kubelet/pods/76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507/volumes"
Jul 13 19:12:52.772099 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:52.772063    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk"]
Jul 13 19:12:52.781987 ip-10-0-139-134 kubenswrapper[2569]: E0713 19:12:52.781946    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 19:12:52.781987 ip-10-0-139-134 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:12:52.781987 ip-10-0-139-134 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 13 19:12:52.781987 ip-10-0-139-134 kubenswrapper[2569]:  > containerID="3d76b5f3342f1594c09dec42e63fdafaa047c1d3b71d60955fca00799c99fe71" cmd=["/bin/sleep","15"]
Jul 13 19:12:52.782320 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:52.782001    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk" message="" err=<
Jul 13 19:12:52.782320 ip-10-0-139-134 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:12:52.782320 ip-10-0-139-134 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 13 19:12:52.782320 ip-10-0-139-134 kubenswrapper[2569]:  >
Jul 13 19:12:52.782320 ip-10-0-139-134 kubenswrapper[2569]: E0713 19:12:52.782016    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 19:12:52.782320 ip-10-0-139-134 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:12:52.782320 ip-10-0-139-134 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 13 19:12:52.782320 ip-10-0-139-134 kubenswrapper[2569]:  > pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk" podUID="6dca69aa-72ad-4a4c-97ff-9b0681efd4ed" containerName="main" containerID="cri-o://3d76b5f3342f1594c09dec42e63fdafaa047c1d3b71d60955fca00799c99fe71"
Jul 13 19:12:52.782320 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:52.782051    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk" podUID="6dca69aa-72ad-4a4c-97ff-9b0681efd4ed" containerName="main" containerID="cri-o://3d76b5f3342f1594c09dec42e63fdafaa047c1d3b71d60955fca00799c99fe71" gracePeriod=60
Jul 13 19:12:53.048427 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:53.048399    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk"
Jul 13 19:12:53.124414 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:53.124374    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6dca69aa-72ad-4a4c-97ff-9b0681efd4ed-tokenizer-uds\") pod \"6dca69aa-72ad-4a4c-97ff-9b0681efd4ed\" (UID: \"6dca69aa-72ad-4a4c-97ff-9b0681efd4ed\") "
Jul 13 19:12:53.124623 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:53.124441    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch5rj\" (UniqueName: \"kubernetes.io/projected/6dca69aa-72ad-4a4c-97ff-9b0681efd4ed-kube-api-access-ch5rj\") pod \"6dca69aa-72ad-4a4c-97ff-9b0681efd4ed\" (UID: \"6dca69aa-72ad-4a4c-97ff-9b0681efd4ed\") "
Jul 13 19:12:53.124623 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:53.124512    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dca69aa-72ad-4a4c-97ff-9b0681efd4ed-tls-certs\") pod \"6dca69aa-72ad-4a4c-97ff-9b0681efd4ed\" (UID: \"6dca69aa-72ad-4a4c-97ff-9b0681efd4ed\") "
Jul 13 19:12:53.124732 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:53.124664    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dca69aa-72ad-4a4c-97ff-9b0681efd4ed-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "6dca69aa-72ad-4a4c-97ff-9b0681efd4ed" (UID: "6dca69aa-72ad-4a4c-97ff-9b0681efd4ed"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:12:53.124827 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:53.124810    2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6dca69aa-72ad-4a4c-97ff-9b0681efd4ed-tokenizer-uds\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:12:53.126884 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:53.126856    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dca69aa-72ad-4a4c-97ff-9b0681efd4ed-kube-api-access-ch5rj" (OuterVolumeSpecName: "kube-api-access-ch5rj") pod "6dca69aa-72ad-4a4c-97ff-9b0681efd4ed" (UID: "6dca69aa-72ad-4a4c-97ff-9b0681efd4ed"). InnerVolumeSpecName "kube-api-access-ch5rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:12:53.127067 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:53.127042    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dca69aa-72ad-4a4c-97ff-9b0681efd4ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6dca69aa-72ad-4a4c-97ff-9b0681efd4ed" (UID: "6dca69aa-72ad-4a4c-97ff-9b0681efd4ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:12:53.225393 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:53.225352    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch5rj\" (UniqueName: \"kubernetes.io/projected/6dca69aa-72ad-4a4c-97ff-9b0681efd4ed-kube-api-access-ch5rj\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:12:53.225393 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:53.225381    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dca69aa-72ad-4a4c-97ff-9b0681efd4ed-tls-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:12:53.280355 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:53.280252    2569 generic.go:356] "Generic (PLEG): container finished" podID="6dca69aa-72ad-4a4c-97ff-9b0681efd4ed" containerID="3d76b5f3342f1594c09dec42e63fdafaa047c1d3b71d60955fca00799c99fe71" exitCode=0
Jul 13 19:12:53.280355 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:53.280311    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk"
Jul 13 19:12:53.280355 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:53.280329    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk" event={"ID":"6dca69aa-72ad-4a4c-97ff-9b0681efd4ed","Type":"ContainerDied","Data":"3d76b5f3342f1594c09dec42e63fdafaa047c1d3b71d60955fca00799c99fe71"}
Jul 13 19:12:53.280630 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:53.280393    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk" event={"ID":"6dca69aa-72ad-4a4c-97ff-9b0681efd4ed","Type":"ContainerDied","Data":"9e515f9a1f2b333dcaf3d9e2ea0abeab9b3822e07a03cd8df8c17c8dc8479425"}
Jul 13 19:12:53.280630 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:53.280421    2569 scope.go:117] "RemoveContainer" containerID="3d76b5f3342f1594c09dec42e63fdafaa047c1d3b71d60955fca00799c99fe71"
Jul 13 19:12:53.289737 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:53.289715    2569 scope.go:117] "RemoveContainer" containerID="3d76b5f3342f1594c09dec42e63fdafaa047c1d3b71d60955fca00799c99fe71"
Jul 13 19:12:53.290067 ip-10-0-139-134 kubenswrapper[2569]: E0713 19:12:53.290040    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d76b5f3342f1594c09dec42e63fdafaa047c1d3b71d60955fca00799c99fe71\": container with ID starting with 3d76b5f3342f1594c09dec42e63fdafaa047c1d3b71d60955fca00799c99fe71 not found: ID does not exist" containerID="3d76b5f3342f1594c09dec42e63fdafaa047c1d3b71d60955fca00799c99fe71"
Jul 13 19:12:53.290169 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:53.290073    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d76b5f3342f1594c09dec42e63fdafaa047c1d3b71d60955fca00799c99fe71"} err="failed to get container status \"3d76b5f3342f1594c09dec42e63fdafaa047c1d3b71d60955fca00799c99fe71\": rpc error: code = NotFound desc = could not find container \"3d76b5f3342f1594c09dec42e63fdafaa047c1d3b71d60955fca00799c99fe71\": container with ID starting with 3d76b5f3342f1594c09dec42e63fdafaa047c1d3b71d60955fca00799c99fe71 not found: ID does not exist"
Jul 13 19:12:53.334057 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:53.334027    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk"]
Jul 13 19:12:53.358772 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:53.358739    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk"]
Jul 13 19:12:55.061037 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:12:55.060990    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dca69aa-72ad-4a4c-97ff-9b0681efd4ed" path="/var/lib/kubelet/pods/6dca69aa-72ad-4a4c-97ff-9b0681efd4ed/volumes"
Jul 13 19:14:07.957960 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:07.957918    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"]
Jul 13 19:14:07.958451 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:07.958251    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dca69aa-72ad-4a4c-97ff-9b0681efd4ed" containerName="main"
Jul 13 19:14:07.958451 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:07.958262    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dca69aa-72ad-4a4c-97ff-9b0681efd4ed" containerName="main"
Jul 13 19:14:07.958451 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:07.958300    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507" containerName="main"
Jul 13 19:14:07.958451 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:07.958307    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507" containerName="main"
Jul 13 19:14:07.958451 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:07.958392    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dca69aa-72ad-4a4c-97ff-9b0681efd4ed" containerName="main"
Jul 13 19:14:07.958451 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:07.958404    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="76d0d9ac-8cf7-4d65-b4cf-1438a4fdb507" containerName="main"
Jul 13 19:14:07.961581 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:07.961558    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:07.975156 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:07.975116    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"]
Jul 13 19:14:07.992480 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:07.992439    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:07.992480 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:07.992481    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:07.992771 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:07.992498    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee548cde-2276-4296-b6c7-d654c765168c-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:07.992771 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:07.992516    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:07.992771 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:07.992623    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee548cde-2276-4296-b6c7-d654c765168c-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:07.992771 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:07.992683    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee548cde-2276-4296-b6c7-d654c765168c-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:07.992771 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:07.992753    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:07.993038 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:07.992794    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2psw\" (UniqueName: \"kubernetes.io/projected/ee548cde-2276-4296-b6c7-d654c765168c-kube-api-access-t2psw\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:07.993038 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:07.992849    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:08.093793 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.093748    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:08.094015 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.093800    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2psw\" (UniqueName: \"kubernetes.io/projected/ee548cde-2276-4296-b6c7-d654c765168c-kube-api-access-t2psw\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:08.094015 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.093835    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:08.094015 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.093861    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:08.094015 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.093881    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:08.094015 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.093906    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee548cde-2276-4296-b6c7-d654c765168c-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:08.094015 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.093930    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:08.094339 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.094214    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee548cde-2276-4296-b6c7-d654c765168c-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:08.094339 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.094271    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:08.094339 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.094282    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee548cde-2276-4296-b6c7-d654c765168c-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:08.094339 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.094332    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:08.094574 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.094391    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:08.094574 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.094429    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:08.094871 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.094845    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee548cde-2276-4296-b6c7-d654c765168c-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:08.096916 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.096889    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:08.097096 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.097077    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee548cde-2276-4296-b6c7-d654c765168c-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:08.103785 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.103755    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee548cde-2276-4296-b6c7-d654c765168c-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:08.103937 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.103902    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2psw\" (UniqueName: \"kubernetes.io/projected/ee548cde-2276-4296-b6c7-d654c765168c-kube-api-access-t2psw\") pod \"router-gateway-1-openshift-default-799f46c59b-s6nj7\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:08.272116 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.272027    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:08.417832 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.417804    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"]
Jul 13 19:14:08.419636 ip-10-0-139-134 kubenswrapper[2569]: W0713 19:14:08.419609    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee548cde_2276_4296_b6c7_d654c765168c.slice/crio-3611839def8d674853938a9777935df5a8276d6888951ec2ab9f2e9a70f68f0c WatchSource:0}: Error finding container 3611839def8d674853938a9777935df5a8276d6888951ec2ab9f2e9a70f68f0c: Status 404 returned error can't find the container with id 3611839def8d674853938a9777935df5a8276d6888951ec2ab9f2e9a70f68f0c
Jul 13 19:14:08.422024 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.421990    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Jul 13 19:14:08.422101 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.422067    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Jul 13 19:14:08.422137 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.422099    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Jul 13 19:14:08.625079 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.625043    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7" event={"ID":"ee548cde-2276-4296-b6c7-d654c765168c","Type":"ContainerStarted","Data":"bb4aa83db48c0e23fbb916ffbfe3e2d020da026e97c8f3ad9a4449cc3fd83f81"}
Jul 13 19:14:08.625278 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.625097    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7" event={"ID":"ee548cde-2276-4296-b6c7-d654c765168c","Type":"ContainerStarted","Data":"3611839def8d674853938a9777935df5a8276d6888951ec2ab9f2e9a70f68f0c"}
Jul 13 19:14:08.651410 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.651355    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7" podStartSLOduration=1.651337687 podStartE2EDuration="1.651337687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:14:08.649811133 +0000 UTC m=+1470.163610300" watchObservedRunningTime="2026-07-13 19:14:08.651337687 +0000 UTC m=+1470.165136853"
Jul 13 19:14:08.710474 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:08.710441    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"]
Jul 13 19:14:09.272668 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:09.272622    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:09.277802 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:09.277776    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:09.631357 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:09.631326    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:09.632450 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:09.632429    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:10.636085 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:10.636039    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7" podUID="ee548cde-2276-4296-b6c7-d654c765168c" containerName="istio-proxy" containerID="cri-o://bb4aa83db48c0e23fbb916ffbfe3e2d020da026e97c8f3ad9a4449cc3fd83f81" gracePeriod=30
Jul 13 19:14:15.678597 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.678562    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"]
Jul 13 19:14:15.682511 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.682483    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:15.686607 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.686579    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Jul 13 19:14:15.686873 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.686646    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gv2sm\""
Jul 13 19:14:15.691651 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.691625    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"]
Jul 13 19:14:15.766606 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.766575    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj6qf\" (UniqueName: \"kubernetes.io/projected/f0d92ecb-bef4-4ca7-896f-d06add699a76-kube-api-access-qj6qf\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:15.766768 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.766624    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:15.766768 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.766643    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:15.766768 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.766720    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:15.766877 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.766766    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:15.766877 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.766791    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d92ecb-bef4-4ca7-896f-d06add699a76-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:15.867254 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.867219    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:15.867433 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.867259    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d92ecb-bef4-4ca7-896f-d06add699a76-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:15.867433 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.867381    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj6qf\" (UniqueName: \"kubernetes.io/projected/f0d92ecb-bef4-4ca7-896f-d06add699a76-kube-api-access-qj6qf\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:15.867545 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.867444    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:15.867545 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.867468    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:15.867545 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.867505    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:15.867730 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.867700    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:15.867815 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.867793    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:15.867936 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.867913    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:15.870046 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.869987    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:15.870164 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.870076    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d92ecb-bef4-4ca7-896f-d06add699a76-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:15.876144 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.876124    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj6qf\" (UniqueName: \"kubernetes.io/projected/f0d92ecb-bef4-4ca7-896f-d06add699a76-kube-api-access-qj6qf\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:15.895103 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.895038    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:15.968609 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.968567    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-istio-data\") pod \"ee548cde-2276-4296-b6c7-d654c765168c\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") "
Jul 13 19:14:15.968808 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.968626    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-credential-socket\") pod \"ee548cde-2276-4296-b6c7-d654c765168c\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") "
Jul 13 19:14:15.968808 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.968647    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-workload-socket\") pod \"ee548cde-2276-4296-b6c7-d654c765168c\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") "
Jul 13 19:14:15.968808 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.968681    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee548cde-2276-4296-b6c7-d654c765168c-istio-podinfo\") pod \"ee548cde-2276-4296-b6c7-d654c765168c\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") "
Jul 13 19:14:15.968808 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.968710    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee548cde-2276-4296-b6c7-d654c765168c-istiod-ca-cert\") pod \"ee548cde-2276-4296-b6c7-d654c765168c\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") "
Jul 13 19:14:15.968808 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.968733    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2psw\" (UniqueName: \"kubernetes.io/projected/ee548cde-2276-4296-b6c7-d654c765168c-kube-api-access-t2psw\") pod \"ee548cde-2276-4296-b6c7-d654c765168c\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") "
Jul 13 19:14:15.969180 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.969137    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ee548cde-2276-4296-b6c7-d654c765168c" (UID: "ee548cde-2276-4296-b6c7-d654c765168c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:14:15.969270 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.969237    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ee548cde-2276-4296-b6c7-d654c765168c" (UID: "ee548cde-2276-4296-b6c7-d654c765168c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:14:15.969270 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.968787    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee548cde-2276-4296-b6c7-d654c765168c-istio-token\") pod \"ee548cde-2276-4296-b6c7-d654c765168c\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") "
Jul 13 19:14:15.969506 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.969379    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-workload-certs\") pod \"ee548cde-2276-4296-b6c7-d654c765168c\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") "
Jul 13 19:14:15.969506 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.969447    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-istio-envoy\") pod \"ee548cde-2276-4296-b6c7-d654c765168c\" (UID: \"ee548cde-2276-4296-b6c7-d654c765168c\") "
Jul 13 19:14:15.969614 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.969547    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee548cde-2276-4296-b6c7-d654c765168c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ee548cde-2276-4296-b6c7-d654c765168c" (UID: "ee548cde-2276-4296-b6c7-d654c765168c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:14:15.972128 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.969714    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-istio-data" (OuterVolumeSpecName: "istio-data") pod "ee548cde-2276-4296-b6c7-d654c765168c" (UID: "ee548cde-2276-4296-b6c7-d654c765168c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:14:15.972128 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.970259    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ee548cde-2276-4296-b6c7-d654c765168c" (UID: "ee548cde-2276-4296-b6c7-d654c765168c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:14:15.972128 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.971392    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-istio-data\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:14:15.972128 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.971414    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-credential-socket\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:14:15.972128 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.971428    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-workload-socket\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:14:15.972128 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.971443    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee548cde-2276-4296-b6c7-d654c765168c-istiod-ca-cert\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:14:15.972128 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.971461    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-workload-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:14:15.972128 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.972095    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee548cde-2276-4296-b6c7-d654c765168c-kube-api-access-t2psw" (OuterVolumeSpecName: "kube-api-access-t2psw") pod "ee548cde-2276-4296-b6c7-d654c765168c" (UID: "ee548cde-2276-4296-b6c7-d654c765168c"). InnerVolumeSpecName "kube-api-access-t2psw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:14:15.976983 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.973011    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee548cde-2276-4296-b6c7-d654c765168c-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee548cde-2276-4296-b6c7-d654c765168c" (UID: "ee548cde-2276-4296-b6c7-d654c765168c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:14:15.976983 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.973086    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ee548cde-2276-4296-b6c7-d654c765168c" (UID: "ee548cde-2276-4296-b6c7-d654c765168c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:14:15.977159 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:15.977073    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee548cde-2276-4296-b6c7-d654c765168c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ee548cde-2276-4296-b6c7-d654c765168c" (UID: "ee548cde-2276-4296-b6c7-d654c765168c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 13 19:14:16.007101 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:16.007053    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:16.072773 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:16.072740    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee548cde-2276-4296-b6c7-d654c765168c-istio-podinfo\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:14:16.072773 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:16.072773    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2psw\" (UniqueName: \"kubernetes.io/projected/ee548cde-2276-4296-b6c7-d654c765168c-kube-api-access-t2psw\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:14:16.073036 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:16.072783    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee548cde-2276-4296-b6c7-d654c765168c-istio-token\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:14:16.073036 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:16.072792    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee548cde-2276-4296-b6c7-d654c765168c-istio-envoy\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:14:16.150325 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:16.150280    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"]
Jul 13 19:14:16.152058 ip-10-0-139-134 kubenswrapper[2569]: W0713 19:14:16.152031    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d92ecb_bef4_4ca7_896f_d06add699a76.slice/crio-53ff51b7bd2a36ebb0eb54f3d85486b87b50260410fc5d635c048b8ef01445fd WatchSource:0}: Error finding container 53ff51b7bd2a36ebb0eb54f3d85486b87b50260410fc5d635c048b8ef01445fd: Status 404 returned error can't find the container with id 53ff51b7bd2a36ebb0eb54f3d85486b87b50260410fc5d635c048b8ef01445fd
Jul 13 19:14:16.664693 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:16.664657    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm" event={"ID":"f0d92ecb-bef4-4ca7-896f-d06add699a76","Type":"ContainerStarted","Data":"c4a1215de10d9c76d5508854c6ea575fbbb24654d66ce9bac6917be92ba23476"}
Jul 13 19:14:16.664889 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:16.664718    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm" event={"ID":"f0d92ecb-bef4-4ca7-896f-d06add699a76","Type":"ContainerStarted","Data":"53ff51b7bd2a36ebb0eb54f3d85486b87b50260410fc5d635c048b8ef01445fd"}
Jul 13 19:14:16.666073 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:16.666048    2569 generic.go:356] "Generic (PLEG): container finished" podID="ee548cde-2276-4296-b6c7-d654c765168c" containerID="bb4aa83db48c0e23fbb916ffbfe3e2d020da026e97c8f3ad9a4449cc3fd83f81" exitCode=0
Jul 13 19:14:16.666199 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:16.666113    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"
Jul 13 19:14:16.666199 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:16.666133    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7" event={"ID":"ee548cde-2276-4296-b6c7-d654c765168c","Type":"ContainerDied","Data":"bb4aa83db48c0e23fbb916ffbfe3e2d020da026e97c8f3ad9a4449cc3fd83f81"}
Jul 13 19:14:16.666199 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:16.666190    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7" event={"ID":"ee548cde-2276-4296-b6c7-d654c765168c","Type":"ContainerDied","Data":"3611839def8d674853938a9777935df5a8276d6888951ec2ab9f2e9a70f68f0c"}
Jul 13 19:14:16.666380 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:16.666213    2569 scope.go:117] "RemoveContainer" containerID="bb4aa83db48c0e23fbb916ffbfe3e2d020da026e97c8f3ad9a4449cc3fd83f81"
Jul 13 19:14:16.676081 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:16.676056    2569 scope.go:117] "RemoveContainer" containerID="bb4aa83db48c0e23fbb916ffbfe3e2d020da026e97c8f3ad9a4449cc3fd83f81"
Jul 13 19:14:16.676451 ip-10-0-139-134 kubenswrapper[2569]: E0713 19:14:16.676427    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb4aa83db48c0e23fbb916ffbfe3e2d020da026e97c8f3ad9a4449cc3fd83f81\": container with ID starting with bb4aa83db48c0e23fbb916ffbfe3e2d020da026e97c8f3ad9a4449cc3fd83f81 not found: ID does not exist" containerID="bb4aa83db48c0e23fbb916ffbfe3e2d020da026e97c8f3ad9a4449cc3fd83f81"
Jul 13 19:14:16.676525 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:16.676459    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb4aa83db48c0e23fbb916ffbfe3e2d020da026e97c8f3ad9a4449cc3fd83f81"} err="failed to get container status \"bb4aa83db48c0e23fbb916ffbfe3e2d020da026e97c8f3ad9a4449cc3fd83f81\": rpc error: code = NotFound desc = could not find container \"bb4aa83db48c0e23fbb916ffbfe3e2d020da026e97c8f3ad9a4449cc3fd83f81\": container with ID starting with bb4aa83db48c0e23fbb916ffbfe3e2d020da026e97c8f3ad9a4449cc3fd83f81 not found: ID does not exist"
Jul 13 19:14:16.683329 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:16.683258    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm" podStartSLOduration=1.683241564 podStartE2EDuration="1.683241564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:14:16.681405199 +0000 UTC m=+1478.195204364" watchObservedRunningTime="2026-07-13 19:14:16.683241564 +0000 UTC m=+1478.197040729"
Jul 13 19:14:16.695697 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:16.695664    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"]
Jul 13 19:14:16.704210 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:16.704176    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7"]
Jul 13 19:14:17.007569 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:17.007478    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:17.061766 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:17.061734    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee548cde-2276-4296-b6c7-d654c765168c" path="/var/lib/kubelet/pods/ee548cde-2276-4296-b6c7-d654c765168c/volumes"
Jul 13 19:14:26.008091 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:26.008044    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:26.021151 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:26.021112    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:26.716939 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:26.716836    2569 generic.go:356] "Generic (PLEG): container finished" podID="8d041b3e-8b24-4ef1-8550-f814a5d09104" containerID="9c78b563e02a451aeb39bc78ab543daae9bb33fedcb08bb8cb4e4f89285b4148" exitCode=0
Jul 13 19:14:26.716939 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:26.716911    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh" event={"ID":"8d041b3e-8b24-4ef1-8550-f814a5d09104","Type":"ContainerDied","Data":"9c78b563e02a451aeb39bc78ab543daae9bb33fedcb08bb8cb4e4f89285b4148"}
Jul 13 19:14:26.717468 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:26.717448    2569 scope.go:117] "RemoveContainer" containerID="9c78b563e02a451aeb39bc78ab543daae9bb33fedcb08bb8cb4e4f89285b4148"
Jul 13 19:14:26.718391 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:26.718367    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Jul 13 19:14:26.718456 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:26.718432    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Jul 13 19:14:26.718499 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:26.718467    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Jul 13 19:14:26.729415 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:26.729386    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:14:27.723711 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:27.723668    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh" event={"ID":"8d041b3e-8b24-4ef1-8550-f814a5d09104","Type":"ContainerStarted","Data":"a9560b05c32d5b2caac21d0e5e54cf2234f2bc6f25ec9c7ee107bd5967fa32d9"}
Jul 13 19:14:28.177446 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:28.177405    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:14:28.262252 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:28.262211    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:14:28.263488 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:28.263452    2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh" podUID="8d041b3e-8b24-4ef1-8550-f814a5d09104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 13 19:14:29.262062 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:29.262015    2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh" podUID="8d041b3e-8b24-4ef1-8550-f814a5d09104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 13 19:14:30.261843 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:30.261801    2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh" podUID="8d041b3e-8b24-4ef1-8550-f814a5d09104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 13 19:14:31.261696 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:31.261651    2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh" podUID="8d041b3e-8b24-4ef1-8550-f814a5d09104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 13 19:14:32.262159 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:32.262113    2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh" podUID="8d041b3e-8b24-4ef1-8550-f814a5d09104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 13 19:14:33.265999 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:33.265965    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:14:33.267221 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:33.267197    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:14:39.033170 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:39.033141    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:14:39.034377 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:39.034355    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:14:39.037851 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:39.037826    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:14:39.038585 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:14:39.038567    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:16:31.869382 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:31.869280    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"]
Jul 13 19:16:46.890308 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:46.890250    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm" podUID="f0d92ecb-bef4-4ca7-896f-d06add699a76" containerName="main" containerID="cri-o://c4a1215de10d9c76d5508854c6ea575fbbb24654d66ce9bac6917be92ba23476" gracePeriod=45
Jul 13 19:16:47.147132 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.147051    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:16:47.254568 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.254534    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-tmp-dir\") pod \"f0d92ecb-bef4-4ca7-896f-d06add699a76\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") "
Jul 13 19:16:47.254756 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.254595    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-home\") pod \"f0d92ecb-bef4-4ca7-896f-d06add699a76\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") "
Jul 13 19:16:47.254756 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.254628    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-model-cache\") pod \"f0d92ecb-bef4-4ca7-896f-d06add699a76\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") "
Jul 13 19:16:47.254756 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.254666    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d92ecb-bef4-4ca7-896f-d06add699a76-tls-certs\") pod \"f0d92ecb-bef4-4ca7-896f-d06add699a76\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") "
Jul 13 19:16:47.254756 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.254713    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qj6qf\" (UniqueName: \"kubernetes.io/projected/f0d92ecb-bef4-4ca7-896f-d06add699a76-kube-api-access-qj6qf\") pod \"f0d92ecb-bef4-4ca7-896f-d06add699a76\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") "
Jul 13 19:16:47.254756 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.254736    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-dshm\") pod \"f0d92ecb-bef4-4ca7-896f-d06add699a76\" (UID: \"f0d92ecb-bef4-4ca7-896f-d06add699a76\") "
Jul 13 19:16:47.255036 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.254858    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-home" (OuterVolumeSpecName: "home") pod "f0d92ecb-bef4-4ca7-896f-d06add699a76" (UID: "f0d92ecb-bef4-4ca7-896f-d06add699a76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:16:47.255036 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.254862    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f0d92ecb-bef4-4ca7-896f-d06add699a76" (UID: "f0d92ecb-bef4-4ca7-896f-d06add699a76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:16:47.255036 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.254956    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-model-cache" (OuterVolumeSpecName: "model-cache") pod "f0d92ecb-bef4-4ca7-896f-d06add699a76" (UID: "f0d92ecb-bef4-4ca7-896f-d06add699a76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:16:47.255196 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.255096    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-tmp-dir\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:16:47.255196 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.255116    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-home\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:16:47.255196 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.255130    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-model-cache\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:16:47.257075 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.257046    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-dshm" (OuterVolumeSpecName: "dshm") pod "f0d92ecb-bef4-4ca7-896f-d06add699a76" (UID: "f0d92ecb-bef4-4ca7-896f-d06add699a76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:16:47.257280 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.257261    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0d92ecb-bef4-4ca7-896f-d06add699a76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0d92ecb-bef4-4ca7-896f-d06add699a76" (UID: "f0d92ecb-bef4-4ca7-896f-d06add699a76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:16:47.257390 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.257344    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d92ecb-bef4-4ca7-896f-d06add699a76-kube-api-access-qj6qf" (OuterVolumeSpecName: "kube-api-access-qj6qf") pod "f0d92ecb-bef4-4ca7-896f-d06add699a76" (UID: "f0d92ecb-bef4-4ca7-896f-d06add699a76"). InnerVolumeSpecName "kube-api-access-qj6qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:16:47.355589 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.355544    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d92ecb-bef4-4ca7-896f-d06add699a76-tls-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:16:47.355589 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.355584    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj6qf\" (UniqueName: \"kubernetes.io/projected/f0d92ecb-bef4-4ca7-896f-d06add699a76-kube-api-access-qj6qf\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:16:47.355589 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.355597    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d92ecb-bef4-4ca7-896f-d06add699a76-dshm\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:16:47.403155 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.403062    2569 generic.go:356] "Generic (PLEG): container finished" podID="f0d92ecb-bef4-4ca7-896f-d06add699a76" containerID="c4a1215de10d9c76d5508854c6ea575fbbb24654d66ce9bac6917be92ba23476" exitCode=0
Jul 13 19:16:47.403155 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.403117    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"
Jul 13 19:16:47.403155 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.403132    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm" event={"ID":"f0d92ecb-bef4-4ca7-896f-d06add699a76","Type":"ContainerDied","Data":"c4a1215de10d9c76d5508854c6ea575fbbb24654d66ce9bac6917be92ba23476"}
Jul 13 19:16:47.403442 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.403185    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm" event={"ID":"f0d92ecb-bef4-4ca7-896f-d06add699a76","Type":"ContainerDied","Data":"53ff51b7bd2a36ebb0eb54f3d85486b87b50260410fc5d635c048b8ef01445fd"}
Jul 13 19:16:47.403442 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.403208    2569 scope.go:117] "RemoveContainer" containerID="c4a1215de10d9c76d5508854c6ea575fbbb24654d66ce9bac6917be92ba23476"
Jul 13 19:16:47.413174 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.413133    2569 scope.go:117] "RemoveContainer" containerID="c4a1215de10d9c76d5508854c6ea575fbbb24654d66ce9bac6917be92ba23476"
Jul 13 19:16:47.413502 ip-10-0-139-134 kubenswrapper[2569]: E0713 19:16:47.413483    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a1215de10d9c76d5508854c6ea575fbbb24654d66ce9bac6917be92ba23476\": container with ID starting with c4a1215de10d9c76d5508854c6ea575fbbb24654d66ce9bac6917be92ba23476 not found: ID does not exist" containerID="c4a1215de10d9c76d5508854c6ea575fbbb24654d66ce9bac6917be92ba23476"
Jul 13 19:16:47.413564 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.413513    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a1215de10d9c76d5508854c6ea575fbbb24654d66ce9bac6917be92ba23476"} err="failed to get container status \"c4a1215de10d9c76d5508854c6ea575fbbb24654d66ce9bac6917be92ba23476\": rpc error: code = NotFound desc = could not find container \"c4a1215de10d9c76d5508854c6ea575fbbb24654d66ce9bac6917be92ba23476\": container with ID starting with c4a1215de10d9c76d5508854c6ea575fbbb24654d66ce9bac6917be92ba23476 not found: ID does not exist"
Jul 13 19:16:47.427326 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.427263    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"]
Jul 13 19:16:47.432203 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:47.432169    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm"]
Jul 13 19:16:49.061336 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:16:49.061279    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0d92ecb-bef4-4ca7-896f-d06add699a76" path="/var/lib/kubelet/pods/f0d92ecb-bef4-4ca7-896f-d06add699a76/volumes"
Jul 13 19:17:52.032512 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.032478    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff"]
Jul 13 19:17:52.032914 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.032888    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee548cde-2276-4296-b6c7-d654c765168c" containerName="istio-proxy"
Jul 13 19:17:52.032914 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.032900    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee548cde-2276-4296-b6c7-d654c765168c" containerName="istio-proxy"
Jul 13 19:17:52.032914 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.032912    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0d92ecb-bef4-4ca7-896f-d06add699a76" containerName="main"
Jul 13 19:17:52.033019 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.032918    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0d92ecb-bef4-4ca7-896f-d06add699a76" containerName="main"
Jul 13 19:17:52.033019 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.032976    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee548cde-2276-4296-b6c7-d654c765168c" containerName="istio-proxy"
Jul 13 19:17:52.033019 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.032984    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0d92ecb-bef4-4ca7-896f-d06add699a76" containerName="main"
Jul 13 19:17:52.037593 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.037570    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff"
Jul 13 19:17:52.041997 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.041957    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gv2sm\""
Jul 13 19:17:52.041997 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.041972    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 13 19:17:52.042271 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.041973    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-d8pk7\""
Jul 13 19:17:52.044761 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.044730    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff"]
Jul 13 19:17:52.139937 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.139899    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff\" (UID: \"42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff"
Jul 13 19:17:52.140134 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.139975    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgccc\" (UniqueName: \"kubernetes.io/projected/42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446-kube-api-access-zgccc\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff\" (UID: \"42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff"
Jul 13 19:17:52.140134 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.140011    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff\" (UID: \"42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff"
Jul 13 19:17:52.241476 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.241437    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff\" (UID: \"42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff"
Jul 13 19:17:52.241679 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.241517    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgccc\" (UniqueName: \"kubernetes.io/projected/42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446-kube-api-access-zgccc\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff\" (UID: \"42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff"
Jul 13 19:17:52.241679 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.241546    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff\" (UID: \"42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff"
Jul 13 19:17:52.241906 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.241884    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff\" (UID: \"42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff"
Jul 13 19:17:52.244083 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.244060    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff\" (UID: \"42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff"
Jul 13 19:17:52.250435 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.250411    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgccc\" (UniqueName: \"kubernetes.io/projected/42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446-kube-api-access-zgccc\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff\" (UID: \"42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff"
Jul 13 19:17:52.348399 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.348360    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff"
Jul 13 19:17:52.489819 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.489793    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff"]
Jul 13 19:17:52.492245 ip-10-0-139-134 kubenswrapper[2569]: W0713 19:17:52.492221    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f8b4d4_e92d_4c0a_8f47_06c3ec4ad446.slice/crio-20e45e93c33e7cb90e04ec449832a0407ef1749c0d252facf2af2fe6f08aa6ec WatchSource:0}: Error finding container 20e45e93c33e7cb90e04ec449832a0407ef1749c0d252facf2af2fe6f08aa6ec: Status 404 returned error can't find the container with id 20e45e93c33e7cb90e04ec449832a0407ef1749c0d252facf2af2fe6f08aa6ec
Jul 13 19:17:52.700010 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.699900    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff" event={"ID":"42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446","Type":"ContainerStarted","Data":"315e85ec70223bfce2eeaa66dcdb3e7c586c1cde9778a18356dc1ab7e750c808"}
Jul 13 19:17:52.700010 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.699956    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff" event={"ID":"42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446","Type":"ContainerStarted","Data":"20e45e93c33e7cb90e04ec449832a0407ef1749c0d252facf2af2fe6f08aa6ec"}
Jul 13 19:17:52.700010 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.699985    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff"
Jul 13 19:17:52.742918 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:17:52.742847    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff" podStartSLOduration=0.742829603 podStartE2EDuration="742.829603ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 19:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:17:52.741730388 +0000 UTC m=+1694.255529556" watchObservedRunningTime="2026-07-13 19:17:52.742829603 +0000 UTC m=+1694.256628769"
Jul 13 19:18:23.707223 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:18:23.707191    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff"
Jul 13 19:18:39.139837 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:18:39.139803    2569 scope.go:117] "RemoveContainer" containerID="9ffe476b1b398071ab3bb5e6f0c2e6148f9d64ebdc9ea8c4f973719ff21fdfe0"
Jul 13 19:19:39.063348 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:19:39.063318    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:19:39.064005 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:19:39.063982    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:19:39.067669 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:19:39.067641    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:19:39.068068 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:19:39.068053    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:22:08.438840 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:08.438805    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"]
Jul 13 19:22:08.439447 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:08.439075    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh" podUID="8d041b3e-8b24-4ef1-8550-f814a5d09104" containerName="istio-proxy" containerID="cri-o://a9560b05c32d5b2caac21d0e5e54cf2234f2bc6f25ec9c7ee107bd5967fa32d9" gracePeriod=30
Jul 13 19:22:13.692487 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.692460    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:22:13.774852 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.774755    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d041b3e-8b24-4ef1-8550-f814a5d09104-istiod-ca-cert\") pod \"8d041b3e-8b24-4ef1-8550-f814a5d09104\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") "
Jul 13 19:22:13.774852 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.774801    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-podinfo\") pod \"8d041b3e-8b24-4ef1-8550-f814a5d09104\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") "
Jul 13 19:22:13.774852 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.774820    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-workload-certs\") pod \"8d041b3e-8b24-4ef1-8550-f814a5d09104\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") "
Jul 13 19:22:13.774852 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.774856    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-credential-socket\") pod \"8d041b3e-8b24-4ef1-8550-f814a5d09104\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") "
Jul 13 19:22:13.775155 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.774883    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-data\") pod \"8d041b3e-8b24-4ef1-8550-f814a5d09104\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") "
Jul 13 19:22:13.775155 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.774907    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-workload-socket\") pod \"8d041b3e-8b24-4ef1-8550-f814a5d09104\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") "
Jul 13 19:22:13.775155 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.774958    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6s6z\" (UniqueName: \"kubernetes.io/projected/8d041b3e-8b24-4ef1-8550-f814a5d09104-kube-api-access-k6s6z\") pod \"8d041b3e-8b24-4ef1-8550-f814a5d09104\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") "
Jul 13 19:22:13.775155 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.774985    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-token\") pod \"8d041b3e-8b24-4ef1-8550-f814a5d09104\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") "
Jul 13 19:22:13.775155 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.775021    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-envoy\") pod \"8d041b3e-8b24-4ef1-8550-f814a5d09104\" (UID: \"8d041b3e-8b24-4ef1-8550-f814a5d09104\") "
Jul 13 19:22:13.775448 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.775171    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d041b3e-8b24-4ef1-8550-f814a5d09104-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8d041b3e-8b24-4ef1-8550-f814a5d09104" (UID: "8d041b3e-8b24-4ef1-8550-f814a5d09104"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:22:13.775448 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.775274    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d041b3e-8b24-4ef1-8550-f814a5d09104-istiod-ca-cert\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:22:13.775448 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.775283    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8d041b3e-8b24-4ef1-8550-f814a5d09104" (UID: "8d041b3e-8b24-4ef1-8550-f814a5d09104"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:22:13.775448 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.775326    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8d041b3e-8b24-4ef1-8550-f814a5d09104" (UID: "8d041b3e-8b24-4ef1-8550-f814a5d09104"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:22:13.775448 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.775425    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8d041b3e-8b24-4ef1-8550-f814a5d09104" (UID: "8d041b3e-8b24-4ef1-8550-f814a5d09104"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:22:13.775773 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.775630    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-data" (OuterVolumeSpecName: "istio-data") pod "8d041b3e-8b24-4ef1-8550-f814a5d09104" (UID: "8d041b3e-8b24-4ef1-8550-f814a5d09104"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:22:13.777248 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.777216    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8d041b3e-8b24-4ef1-8550-f814a5d09104" (UID: "8d041b3e-8b24-4ef1-8550-f814a5d09104"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 13 19:22:13.777442 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.777410    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d041b3e-8b24-4ef1-8550-f814a5d09104-kube-api-access-k6s6z" (OuterVolumeSpecName: "kube-api-access-k6s6z") pod "8d041b3e-8b24-4ef1-8550-f814a5d09104" (UID: "8d041b3e-8b24-4ef1-8550-f814a5d09104"). InnerVolumeSpecName "kube-api-access-k6s6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:22:13.777746 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.777727    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8d041b3e-8b24-4ef1-8550-f814a5d09104" (UID: "8d041b3e-8b24-4ef1-8550-f814a5d09104"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:22:13.777813 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.777740    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-token" (OuterVolumeSpecName: "istio-token") pod "8d041b3e-8b24-4ef1-8550-f814a5d09104" (UID: "8d041b3e-8b24-4ef1-8550-f814a5d09104"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:22:13.876057 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.876016    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-data\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:22:13.876057 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.876049    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-workload-socket\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:22:13.876057 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.876063    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6s6z\" (UniqueName: \"kubernetes.io/projected/8d041b3e-8b24-4ef1-8550-f814a5d09104-kube-api-access-k6s6z\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:22:13.876336 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.876075    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-token\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:22:13.876336 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.876087    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-envoy\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:22:13.876336 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.876098    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d041b3e-8b24-4ef1-8550-f814a5d09104-istio-podinfo\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:22:13.876336 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.876108    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-workload-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:22:13.876336 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.876119    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d041b3e-8b24-4ef1-8550-f814a5d09104-credential-socket\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:22:13.930014 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.929979    2569 generic.go:356] "Generic (PLEG): container finished" podID="8d041b3e-8b24-4ef1-8550-f814a5d09104" containerID="a9560b05c32d5b2caac21d0e5e54cf2234f2bc6f25ec9c7ee107bd5967fa32d9" exitCode=0
Jul 13 19:22:13.930214 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.930037    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"
Jul 13 19:22:13.930214 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.930056    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh" event={"ID":"8d041b3e-8b24-4ef1-8550-f814a5d09104","Type":"ContainerDied","Data":"a9560b05c32d5b2caac21d0e5e54cf2234f2bc6f25ec9c7ee107bd5967fa32d9"}
Jul 13 19:22:13.930214 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.930123    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh" event={"ID":"8d041b3e-8b24-4ef1-8550-f814a5d09104","Type":"ContainerDied","Data":"ea9e9ecbc1bf24fb73e6293344e81cc075a6c9e423d4f20c5c26df73d80733ee"}
Jul 13 19:22:13.930214 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.930155    2569 scope.go:117] "RemoveContainer" containerID="a9560b05c32d5b2caac21d0e5e54cf2234f2bc6f25ec9c7ee107bd5967fa32d9"
Jul 13 19:22:13.939435 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.939398    2569 scope.go:117] "RemoveContainer" containerID="9c78b563e02a451aeb39bc78ab543daae9bb33fedcb08bb8cb4e4f89285b4148"
Jul 13 19:22:13.947716 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.947697    2569 scope.go:117] "RemoveContainer" containerID="a9560b05c32d5b2caac21d0e5e54cf2234f2bc6f25ec9c7ee107bd5967fa32d9"
Jul 13 19:22:13.948005 ip-10-0-139-134 kubenswrapper[2569]: E0713 19:22:13.947986    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9560b05c32d5b2caac21d0e5e54cf2234f2bc6f25ec9c7ee107bd5967fa32d9\": container with ID starting with a9560b05c32d5b2caac21d0e5e54cf2234f2bc6f25ec9c7ee107bd5967fa32d9 not found: ID does not exist" containerID="a9560b05c32d5b2caac21d0e5e54cf2234f2bc6f25ec9c7ee107bd5967fa32d9"
Jul 13 19:22:13.948053 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.948017    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9560b05c32d5b2caac21d0e5e54cf2234f2bc6f25ec9c7ee107bd5967fa32d9"} err="failed to get container status \"a9560b05c32d5b2caac21d0e5e54cf2234f2bc6f25ec9c7ee107bd5967fa32d9\": rpc error: code = NotFound desc = could not find container \"a9560b05c32d5b2caac21d0e5e54cf2234f2bc6f25ec9c7ee107bd5967fa32d9\": container with ID starting with a9560b05c32d5b2caac21d0e5e54cf2234f2bc6f25ec9c7ee107bd5967fa32d9 not found: ID does not exist"
Jul 13 19:22:13.948053 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.948037    2569 scope.go:117] "RemoveContainer" containerID="9c78b563e02a451aeb39bc78ab543daae9bb33fedcb08bb8cb4e4f89285b4148"
Jul 13 19:22:13.948278 ip-10-0-139-134 kubenswrapper[2569]: E0713 19:22:13.948263    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c78b563e02a451aeb39bc78ab543daae9bb33fedcb08bb8cb4e4f89285b4148\": container with ID starting with 9c78b563e02a451aeb39bc78ab543daae9bb33fedcb08bb8cb4e4f89285b4148 not found: ID does not exist" containerID="9c78b563e02a451aeb39bc78ab543daae9bb33fedcb08bb8cb4e4f89285b4148"
Jul 13 19:22:13.948338 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.948283    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c78b563e02a451aeb39bc78ab543daae9bb33fedcb08bb8cb4e4f89285b4148"} err="failed to get container status \"9c78b563e02a451aeb39bc78ab543daae9bb33fedcb08bb8cb4e4f89285b4148\": rpc error: code = NotFound desc = could not find container \"9c78b563e02a451aeb39bc78ab543daae9bb33fedcb08bb8cb4e4f89285b4148\": container with ID starting with 9c78b563e02a451aeb39bc78ab543daae9bb33fedcb08bb8cb4e4f89285b4148 not found: ID does not exist"
Jul 13 19:22:13.954900 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.954873    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"]
Jul 13 19:22:13.964535 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:13.964507    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh"]
Jul 13 19:22:15.060830 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:22:15.060800    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d041b3e-8b24-4ef1-8550-f814a5d09104" path="/var/lib/kubelet/pods/8d041b3e-8b24-4ef1-8550-f814a5d09104/volumes"
Jul 13 19:24:39.091845 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:24:39.091817    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:24:39.094359 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:24:39.091817    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:24:39.096137 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:24:39.096116    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:24:39.096392 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:24:39.096377    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:29:39.118974 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:29:39.118855    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:29:39.122214 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:29:39.120239    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:29:39.123244 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:29:39.123224    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:29:39.124813 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:29:39.124791    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:32:51.474657 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:32:51.474627    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff_42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446/main/0.log"
Jul 13 19:34:39.147410 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:34:39.147274    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:34:39.150865 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:34:39.147987    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:34:39.151589 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:34:39.151570    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:34:39.152270 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:34:39.152249    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:39:39.174434 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:39:39.174322    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:39:39.180905 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:39:39.179417    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:39:39.181424 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:39:39.181402    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:39:39.183673 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:39:39.183651    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:44:39.203827 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:44:39.203717    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:44:39.208018 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:44:39.207577    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:44:39.208018 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:44:39.207970    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:44:39.211967 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:44:39.211945    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:48:21.618050 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:21.617772    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2"]
Jul 13 19:48:21.618696 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:21.618224    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d041b3e-8b24-4ef1-8550-f814a5d09104" containerName="istio-proxy"
Jul 13 19:48:21.618696 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:21.618242    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d041b3e-8b24-4ef1-8550-f814a5d09104" containerName="istio-proxy"
Jul 13 19:48:21.618696 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:21.618274    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d041b3e-8b24-4ef1-8550-f814a5d09104" containerName="istio-proxy"
Jul 13 19:48:21.618696 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:21.618280    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d041b3e-8b24-4ef1-8550-f814a5d09104" containerName="istio-proxy"
Jul 13 19:48:21.618696 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:21.618371    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d041b3e-8b24-4ef1-8550-f814a5d09104" containerName="istio-proxy"
Jul 13 19:48:21.618696 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:21.618381    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d041b3e-8b24-4ef1-8550-f814a5d09104" containerName="istio-proxy"
Jul 13 19:48:21.621682 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:21.621655    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2"
Jul 13 19:48:21.625320 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:21.625266    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-hnhlw\""
Jul 13 19:48:21.626525 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:21.626504    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 13 19:48:21.633865 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:21.633838    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2"]
Jul 13 19:48:21.727131 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:21.727093    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2ce120d6-0e30-45b7-90fc-f7f76654786e-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2\" (UID: \"2ce120d6-0e30-45b7-90fc-f7f76654786e\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2"
Jul 13 19:48:21.727131 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:21.727137    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ce120d6-0e30-45b7-90fc-f7f76654786e-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2\" (UID: \"2ce120d6-0e30-45b7-90fc-f7f76654786e\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2"
Jul 13 19:48:21.727405 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:21.727320    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrxgx\" (UniqueName: \"kubernetes.io/projected/2ce120d6-0e30-45b7-90fc-f7f76654786e-kube-api-access-hrxgx\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2\" (UID: \"2ce120d6-0e30-45b7-90fc-f7f76654786e\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2"
Jul 13 19:48:21.828886 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:21.828836    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2ce120d6-0e30-45b7-90fc-f7f76654786e-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2\" (UID: \"2ce120d6-0e30-45b7-90fc-f7f76654786e\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2"
Jul 13 19:48:21.828886 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:21.828892    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ce120d6-0e30-45b7-90fc-f7f76654786e-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2\" (UID: \"2ce120d6-0e30-45b7-90fc-f7f76654786e\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2"
Jul 13 19:48:21.829169 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:21.828993    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrxgx\" (UniqueName: \"kubernetes.io/projected/2ce120d6-0e30-45b7-90fc-f7f76654786e-kube-api-access-hrxgx\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2\" (UID: \"2ce120d6-0e30-45b7-90fc-f7f76654786e\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2"
Jul 13 19:48:21.829269 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:21.829247    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2ce120d6-0e30-45b7-90fc-f7f76654786e-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2\" (UID: \"2ce120d6-0e30-45b7-90fc-f7f76654786e\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2"
Jul 13 19:48:21.831638 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:21.831609    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ce120d6-0e30-45b7-90fc-f7f76654786e-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2\" (UID: \"2ce120d6-0e30-45b7-90fc-f7f76654786e\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2"
Jul 13 19:48:21.840473 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:21.840443    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrxgx\" (UniqueName: \"kubernetes.io/projected/2ce120d6-0e30-45b7-90fc-f7f76654786e-kube-api-access-hrxgx\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2\" (UID: \"2ce120d6-0e30-45b7-90fc-f7f76654786e\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2"
Jul 13 19:48:21.933184 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:21.933071    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2"
Jul 13 19:48:22.080512 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:22.080486    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2"]
Jul 13 19:48:22.082672 ip-10-0-139-134 kubenswrapper[2569]: W0713 19:48:22.082639    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce120d6_0e30_45b7_90fc_f7f76654786e.slice/crio-79f112ae3faae4de3ab6273cd5386d6b1d4d55fbdfa23d37f20700bcabf624e2 WatchSource:0}: Error finding container 79f112ae3faae4de3ab6273cd5386d6b1d4d55fbdfa23d37f20700bcabf624e2: Status 404 returned error can't find the container with id 79f112ae3faae4de3ab6273cd5386d6b1d4d55fbdfa23d37f20700bcabf624e2
Jul 13 19:48:23.067995 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:23.067956    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2" event={"ID":"2ce120d6-0e30-45b7-90fc-f7f76654786e","Type":"ContainerStarted","Data":"9ec5f95b130b147427c28f79970902aaa9cdc5d762b85f89422eeae59552326b"}
Jul 13 19:48:23.068458 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:23.068006    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2" event={"ID":"2ce120d6-0e30-45b7-90fc-f7f76654786e","Type":"ContainerStarted","Data":"79f112ae3faae4de3ab6273cd5386d6b1d4d55fbdfa23d37f20700bcabf624e2"}
Jul 13 19:48:23.068458 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:23.068088    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2"
Jul 13 19:48:23.088904 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:23.088847    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2" podStartSLOduration=2.088828321 podStartE2EDuration="2.088828321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 19:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:48:23.087868699 +0000 UTC m=+3524.601667882" watchObservedRunningTime="2026-07-13 19:48:23.088828321 +0000 UTC m=+3524.602627487"
Jul 13 19:48:54.074423 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:48:54.074390    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2"
Jul 13 19:49:39.231729 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:49:39.231595    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:49:39.237729 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:49:39.236104    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:49:39.238263 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:49:39.238242    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:49:39.242410 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:49:39.242391    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:50:22.571048 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.571002    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"]
Jul 13 19:50:22.574847 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.574822    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:22.577751 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.577722    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 13 19:50:22.583147 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.582816    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"]
Jul 13 19:50:22.691252 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.691203    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjvqg\" (UniqueName: \"kubernetes.io/projected/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-kube-api-access-qjvqg\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:22.691500 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.691268    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:22.691500 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.691325    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:22.691500 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.691353    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:22.691500 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.691398    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:22.691500 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.691449    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:22.792441 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.792408    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:22.792653 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.792479    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:22.792653 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.792525    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjvqg\" (UniqueName: \"kubernetes.io/projected/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-kube-api-access-qjvqg\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:22.792653 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.792572    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:22.792653 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.792595    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:22.792653 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.792626    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:22.793065 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.793036    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:22.793252 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.793134    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:22.793252 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.793167    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:22.795765 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.795720    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:22.795993 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.795967    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:22.807323 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.807266    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjvqg\" (UniqueName: \"kubernetes.io/projected/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-kube-api-access-qjvqg\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:22.886654 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:22.886612    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:23.034813 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:23.034784    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"]
Jul 13 19:50:23.036994 ip-10-0-139-134 kubenswrapper[2569]: W0713 19:50:23.036945    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d255dc9_1daf_47bd_a6eb_10ed2d60ab7c.slice/crio-a49e8eb17f94f1291be9f90c9b7b156483730d817f2ba67a4553885ce0561aa1 WatchSource:0}: Error finding container a49e8eb17f94f1291be9f90c9b7b156483730d817f2ba67a4553885ce0561aa1: Status 404 returned error can't find the container with id a49e8eb17f94f1291be9f90c9b7b156483730d817f2ba67a4553885ce0561aa1
Jul 13 19:50:23.630442 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:23.630400    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s" event={"ID":"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c","Type":"ContainerStarted","Data":"092782ca22a92a8de6596b3328dbf8f7db1800bff6befe6e1c5fa6550b5b13d4"}
Jul 13 19:50:23.630903 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:23.630467    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s" event={"ID":"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c","Type":"ContainerStarted","Data":"a49e8eb17f94f1291be9f90c9b7b156483730d817f2ba67a4553885ce0561aa1"}
Jul 13 19:50:23.648682 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:23.648626    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s" podStartSLOduration=1.648608588 podStartE2EDuration="1.648608588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:50:23.648087142 +0000 UTC m=+3645.161886309" watchObservedRunningTime="2026-07-13 19:50:23.648608588 +0000 UTC m=+3645.162407753"
Jul 13 19:50:23.886936 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:23.886838    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:32.886843 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:32.886804    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:32.899605 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:32.899570    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:50:33.688059 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:50:33.688026    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:52:56.240151 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:52:56.240013    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"]
Jul 13 19:53:11.262121 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:11.262064    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s" podUID="7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c" containerName="main" containerID="cri-o://092782ca22a92a8de6596b3328dbf8f7db1800bff6befe6e1c5fa6550b5b13d4" gracePeriod=45
Jul 13 19:53:11.417851 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:11.417815    2569 generic.go:356] "Generic (PLEG): container finished" podID="7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c" containerID="092782ca22a92a8de6596b3328dbf8f7db1800bff6befe6e1c5fa6550b5b13d4" exitCode=0
Jul 13 19:53:11.418018 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:11.417857    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s" event={"ID":"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c","Type":"ContainerDied","Data":"092782ca22a92a8de6596b3328dbf8f7db1800bff6befe6e1c5fa6550b5b13d4"}
Jul 13 19:53:11.522400 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:11.522332    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:53:11.594995 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:11.594957    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-dshm\") pod \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") "
Jul 13 19:53:11.595171 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:11.595005    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-model-cache\") pod \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") "
Jul 13 19:53:11.595171 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:11.595030    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-home\") pod \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") "
Jul 13 19:53:11.595171 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:11.595091    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-tmp-dir\") pod \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") "
Jul 13 19:53:11.595171 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:11.595132    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjvqg\" (UniqueName: \"kubernetes.io/projected/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-kube-api-access-qjvqg\") pod \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") "
Jul 13 19:53:11.595374 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:11.595213    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-tls-certs\") pod \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\" (UID: \"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c\") "
Jul 13 19:53:11.595425 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:11.595348    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c" (UID: "7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:53:11.595425 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:11.595371    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c" (UID: "7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:53:11.595425 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:11.595391    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-home" (OuterVolumeSpecName: "home") pod "7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c" (UID: "7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:53:11.595568 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:11.595550    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-model-cache\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:53:11.595627 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:11.595569    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-home\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:53:11.595627 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:11.595596    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-tmp-dir\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:53:11.597503 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:11.597482    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-kube-api-access-qjvqg" (OuterVolumeSpecName: "kube-api-access-qjvqg") pod "7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c" (UID: "7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c"). InnerVolumeSpecName "kube-api-access-qjvqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:53:11.597888 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:11.597862    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c" (UID: "7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:53:11.597996 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:11.597861    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-dshm" (OuterVolumeSpecName: "dshm") pod "7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c" (UID: "7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:53:11.697060 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:11.697007    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-tls-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:53:11.697060 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:11.697051    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-dshm\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:53:11.697060 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:11.697061    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjvqg\" (UniqueName: \"kubernetes.io/projected/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c-kube-api-access-qjvqg\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:53:12.423722 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:12.423690    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"
Jul 13 19:53:12.423722 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:12.423705    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s" event={"ID":"7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c","Type":"ContainerDied","Data":"a49e8eb17f94f1291be9f90c9b7b156483730d817f2ba67a4553885ce0561aa1"}
Jul 13 19:53:12.424208 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:12.423773    2569 scope.go:117] "RemoveContainer" containerID="092782ca22a92a8de6596b3328dbf8f7db1800bff6befe6e1c5fa6550b5b13d4"
Jul 13 19:53:12.446492 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:12.446456    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"]
Jul 13 19:53:12.450475 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:12.450446    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s"]
Jul 13 19:53:13.061906 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:13.061871    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c" path="/var/lib/kubelet/pods/7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c/volumes"
Jul 13 19:53:44.393912 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.393875    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"]
Jul 13 19:53:44.394339 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.394223    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c" containerName="main"
Jul 13 19:53:44.394339 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.394234    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c" containerName="main"
Jul 13 19:53:44.394339 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.394319    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d255dc9-1daf-47bd-a6eb-10ed2d60ab7c" containerName="main"
Jul 13 19:53:44.398540 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.398519    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:44.401483 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.401463    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 13 19:53:44.408962 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.408927    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"]
Jul 13 19:53:44.487982 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.487942    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:44.488211 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.487996    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:44.488211 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.488065    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/159a2464-bdc3-4bc0-a896-12cff608cf8d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:44.488211 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.488107    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmfgt\" (UniqueName: \"kubernetes.io/projected/159a2464-bdc3-4bc0-a896-12cff608cf8d-kube-api-access-jmfgt\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:44.488211 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.488127    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:44.488420 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.488204    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:44.589113 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.589068    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:44.589437 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.589138    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:44.589437 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.589175    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:44.589437 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.589197    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/159a2464-bdc3-4bc0-a896-12cff608cf8d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:44.589437 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.589215    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmfgt\" (UniqueName: \"kubernetes.io/projected/159a2464-bdc3-4bc0-a896-12cff608cf8d-kube-api-access-jmfgt\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:44.589437 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.589367    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:44.589735 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.589524    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:44.589735 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.589585    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:44.589735 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.589719    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:44.591910 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.591873    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:44.592146 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.592128    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/159a2464-bdc3-4bc0-a896-12cff608cf8d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:44.598556 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.598528    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmfgt\" (UniqueName: \"kubernetes.io/projected/159a2464-bdc3-4bc0-a896-12cff608cf8d-kube-api-access-jmfgt\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:44.710598 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.710503    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:44.862410 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:44.862195    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"]
Jul 13 19:53:44.864970 ip-10-0-139-134 kubenswrapper[2569]: W0713 19:53:44.864943    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159a2464_bdc3_4bc0_a896_12cff608cf8d.slice/crio-62eb90e35519aa6ffe6c5fcae126dc9462a34137ec16e789671a557bdb18e33f WatchSource:0}: Error finding container 62eb90e35519aa6ffe6c5fcae126dc9462a34137ec16e789671a557bdb18e33f: Status 404 returned error can't find the container with id 62eb90e35519aa6ffe6c5fcae126dc9462a34137ec16e789671a557bdb18e33f
Jul 13 19:53:45.583803 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:45.583764    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp" event={"ID":"159a2464-bdc3-4bc0-a896-12cff608cf8d","Type":"ContainerStarted","Data":"f0e2da53f914d97ac87e0616169e4d4c2eb43c8b81496331e5db42d4df7e3e04"}
Jul 13 19:53:45.584221 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:45.583822    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp" event={"ID":"159a2464-bdc3-4bc0-a896-12cff608cf8d","Type":"ContainerStarted","Data":"62eb90e35519aa6ffe6c5fcae126dc9462a34137ec16e789671a557bdb18e33f"}
Jul 13 19:53:45.602858 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:45.602806    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp" podStartSLOduration=1.602788186 podStartE2EDuration="1.602788186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:53:45.601576232 +0000 UTC m=+3847.115375417" watchObservedRunningTime="2026-07-13 19:53:45.602788186 +0000 UTC m=+3847.116587351"
Jul 13 19:53:45.711464 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:45.711430    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:54.711387 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:54.711344    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:54.724410 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:54.724380    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:53:55.643216 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:53:55.643186    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:54:39.259877 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:54:39.259753    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:54:39.264458 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:54:39.264435    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:54:39.267756 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:54:39.267727    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:54:39.272167 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:54:39.272140    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:55:53.777788 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:55:53.777702    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"]
Jul 13 19:56:08.798627 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:08.798582    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp" podUID="159a2464-bdc3-4bc0-a896-12cff608cf8d" containerName="main" containerID="cri-o://f0e2da53f914d97ac87e0616169e4d4c2eb43c8b81496331e5db42d4df7e3e04" gracePeriod=45
Jul 13 19:56:09.049167 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.049100    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:56:09.083123 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.083089    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmfgt\" (UniqueName: \"kubernetes.io/projected/159a2464-bdc3-4bc0-a896-12cff608cf8d-kube-api-access-jmfgt\") pod \"159a2464-bdc3-4bc0-a896-12cff608cf8d\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") "
Jul 13 19:56:09.083327 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.083153    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-tmp-dir\") pod \"159a2464-bdc3-4bc0-a896-12cff608cf8d\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") "
Jul 13 19:56:09.083327 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.083170    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-model-cache\") pod \"159a2464-bdc3-4bc0-a896-12cff608cf8d\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") "
Jul 13 19:56:09.083327 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.083195    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/159a2464-bdc3-4bc0-a896-12cff608cf8d-tls-certs\") pod \"159a2464-bdc3-4bc0-a896-12cff608cf8d\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") "
Jul 13 19:56:09.083327 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.083210    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-home\") pod \"159a2464-bdc3-4bc0-a896-12cff608cf8d\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") "
Jul 13 19:56:09.083327 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.083225    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-dshm\") pod \"159a2464-bdc3-4bc0-a896-12cff608cf8d\" (UID: \"159a2464-bdc3-4bc0-a896-12cff608cf8d\") "
Jul 13 19:56:09.083563 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.083478    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-model-cache" (OuterVolumeSpecName: "model-cache") pod "159a2464-bdc3-4bc0-a896-12cff608cf8d" (UID: "159a2464-bdc3-4bc0-a896-12cff608cf8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:56:09.083668 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.083635    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-model-cache\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:56:09.083813 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.083671    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "159a2464-bdc3-4bc0-a896-12cff608cf8d" (UID: "159a2464-bdc3-4bc0-a896-12cff608cf8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:56:09.083813 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.083687    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-home" (OuterVolumeSpecName: "home") pod "159a2464-bdc3-4bc0-a896-12cff608cf8d" (UID: "159a2464-bdc3-4bc0-a896-12cff608cf8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:56:09.085631 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.085597    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-dshm" (OuterVolumeSpecName: "dshm") pod "159a2464-bdc3-4bc0-a896-12cff608cf8d" (UID: "159a2464-bdc3-4bc0-a896-12cff608cf8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:56:09.085791 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.085634    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/159a2464-bdc3-4bc0-a896-12cff608cf8d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "159a2464-bdc3-4bc0-a896-12cff608cf8d" (UID: "159a2464-bdc3-4bc0-a896-12cff608cf8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:56:09.085791 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.085675    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159a2464-bdc3-4bc0-a896-12cff608cf8d-kube-api-access-jmfgt" (OuterVolumeSpecName: "kube-api-access-jmfgt") pod "159a2464-bdc3-4bc0-a896-12cff608cf8d" (UID: "159a2464-bdc3-4bc0-a896-12cff608cf8d"). InnerVolumeSpecName "kube-api-access-jmfgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:56:09.184527 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.184488    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmfgt\" (UniqueName: \"kubernetes.io/projected/159a2464-bdc3-4bc0-a896-12cff608cf8d-kube-api-access-jmfgt\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:56:09.184527 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.184523    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-tmp-dir\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:56:09.184527 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.184533    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/159a2464-bdc3-4bc0-a896-12cff608cf8d-tls-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:56:09.184527 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.184540    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-home\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:56:09.184801 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.184548    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/159a2464-bdc3-4bc0-a896-12cff608cf8d-dshm\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:56:09.256973 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.256924    2569 generic.go:356] "Generic (PLEG): container finished" podID="159a2464-bdc3-4bc0-a896-12cff608cf8d" containerID="f0e2da53f914d97ac87e0616169e4d4c2eb43c8b81496331e5db42d4df7e3e04" exitCode=0
Jul 13 19:56:09.257151 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.256979    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"
Jul 13 19:56:09.257151 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.257015    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp" event={"ID":"159a2464-bdc3-4bc0-a896-12cff608cf8d","Type":"ContainerDied","Data":"f0e2da53f914d97ac87e0616169e4d4c2eb43c8b81496331e5db42d4df7e3e04"}
Jul 13 19:56:09.257151 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.257075    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp" event={"ID":"159a2464-bdc3-4bc0-a896-12cff608cf8d","Type":"ContainerDied","Data":"62eb90e35519aa6ffe6c5fcae126dc9462a34137ec16e789671a557bdb18e33f"}
Jul 13 19:56:09.257151 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.257100    2569 scope.go:117] "RemoveContainer" containerID="f0e2da53f914d97ac87e0616169e4d4c2eb43c8b81496331e5db42d4df7e3e04"
Jul 13 19:56:09.266232 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.266210    2569 scope.go:117] "RemoveContainer" containerID="f0e2da53f914d97ac87e0616169e4d4c2eb43c8b81496331e5db42d4df7e3e04"
Jul 13 19:56:09.266596 ip-10-0-139-134 kubenswrapper[2569]: E0713 19:56:09.266574    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e2da53f914d97ac87e0616169e4d4c2eb43c8b81496331e5db42d4df7e3e04\": container with ID starting with f0e2da53f914d97ac87e0616169e4d4c2eb43c8b81496331e5db42d4df7e3e04 not found: ID does not exist" containerID="f0e2da53f914d97ac87e0616169e4d4c2eb43c8b81496331e5db42d4df7e3e04"
Jul 13 19:56:09.266648 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.266606    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e2da53f914d97ac87e0616169e4d4c2eb43c8b81496331e5db42d4df7e3e04"} err="failed to get container status \"f0e2da53f914d97ac87e0616169e4d4c2eb43c8b81496331e5db42d4df7e3e04\": rpc error: code = NotFound desc = could not find container \"f0e2da53f914d97ac87e0616169e4d4c2eb43c8b81496331e5db42d4df7e3e04\": container with ID starting with f0e2da53f914d97ac87e0616169e4d4c2eb43c8b81496331e5db42d4df7e3e04 not found: ID does not exist"
Jul 13 19:56:09.285672 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.285641    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"]
Jul 13 19:56:09.290746 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:09.290715    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp"]
Jul 13 19:56:11.061492 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:11.061461    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="159a2464-bdc3-4bc0-a896-12cff608cf8d" path="/var/lib/kubelet/pods/159a2464-bdc3-4bc0-a896-12cff608cf8d/volumes"
Jul 13 19:56:58.288482 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.288447    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"]
Jul 13 19:56:58.289101 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.288964    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="159a2464-bdc3-4bc0-a896-12cff608cf8d" containerName="main"
Jul 13 19:56:58.289101 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.288981    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="159a2464-bdc3-4bc0-a896-12cff608cf8d" containerName="main"
Jul 13 19:56:58.289101 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.289051    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="159a2464-bdc3-4bc0-a896-12cff608cf8d" containerName="main"
Jul 13 19:56:58.292011 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.291990    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:56:58.295281 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.295258    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 13 19:56:58.303888 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.303862    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"]
Jul 13 19:56:58.429560 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.429528    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:56:58.429560 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.429573    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:56:58.429806 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.429597    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkdx2\" (UniqueName: \"kubernetes.io/projected/a4b80443-c66f-4ee0-bba7-4e7484f385ec-kube-api-access-vkdx2\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:56:58.429806 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.429670    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:56:58.429806 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.429699    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4b80443-c66f-4ee0-bba7-4e7484f385ec-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:56:58.429806 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.429719    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:56:58.530646 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.530609    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4b80443-c66f-4ee0-bba7-4e7484f385ec-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:56:58.530646 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.530658    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:56:58.530930 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.530711    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:56:58.530930 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.530743    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:56:58.530930 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.530776    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkdx2\" (UniqueName: \"kubernetes.io/projected/a4b80443-c66f-4ee0-bba7-4e7484f385ec-kube-api-access-vkdx2\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:56:58.530930 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.530819    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:56:58.531237 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.531194    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:56:58.531334 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.531240    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:56:58.531334 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.531262    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:56:58.533129 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.533104    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:56:58.533498 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.533481    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4b80443-c66f-4ee0-bba7-4e7484f385ec-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:56:58.539591 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.539531    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkdx2\" (UniqueName: \"kubernetes.io/projected/a4b80443-c66f-4ee0-bba7-4e7484f385ec-kube-api-access-vkdx2\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:56:58.603581 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.603548    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:56:58.749495 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:58.749467    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"]
Jul 13 19:56:58.752195 ip-10-0-139-134 kubenswrapper[2569]: W0713 19:56:58.752100    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b80443_c66f_4ee0_bba7_4e7484f385ec.slice/crio-be6c4950001d30927132ae1d064670f295eeec63966bf8b8e6e8ff2aff01941c WatchSource:0}: Error finding container be6c4950001d30927132ae1d064670f295eeec63966bf8b8e6e8ff2aff01941c: Status 404 returned error can't find the container with id be6c4950001d30927132ae1d064670f295eeec63966bf8b8e6e8ff2aff01941c
Jul 13 19:56:59.480463 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:59.480428    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n" event={"ID":"a4b80443-c66f-4ee0-bba7-4e7484f385ec","Type":"ContainerStarted","Data":"9f79933c25aa21d115d4f56b227f78fa84a737ed26df7efae60cf51bebb97e3e"}
Jul 13 19:56:59.480903 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:59.480487    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n" event={"ID":"a4b80443-c66f-4ee0-bba7-4e7484f385ec","Type":"ContainerStarted","Data":"be6c4950001d30927132ae1d064670f295eeec63966bf8b8e6e8ff2aff01941c"}
Jul 13 19:56:59.515472 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:59.515414    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n" podStartSLOduration=1.515395627 podStartE2EDuration="1.515395627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:56:59.513664578 +0000 UTC m=+4041.027463743" watchObservedRunningTime="2026-07-13 19:56:59.515395627 +0000 UTC m=+4041.029194793"
Jul 13 19:56:59.604390 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:56:59.604344    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:57:08.604649 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:57:08.604601    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:57:08.617480 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:57:08.617450    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:57:09.540412 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:57:09.540384    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:59:08.386361 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:08.386324    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"]
Jul 13 19:59:23.408243 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:23.408183    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n" podUID="a4b80443-c66f-4ee0-bba7-4e7484f385ec" containerName="main" containerID="cri-o://9f79933c25aa21d115d4f56b227f78fa84a737ed26df7efae60cf51bebb97e3e" gracePeriod=45
Jul 13 19:59:23.529878 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:23.529836    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n" podUID="a4b80443-c66f-4ee0-bba7-4e7484f385ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 13 19:59:23.668506 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:23.668434    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:59:23.724011 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:23.723981    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4b80443-c66f-4ee0-bba7-4e7484f385ec-tls-certs\") pod \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") "
Jul 13 19:59:23.724208 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:23.724031    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-home\") pod \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") "
Jul 13 19:59:23.724208 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:23.724075    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkdx2\" (UniqueName: \"kubernetes.io/projected/a4b80443-c66f-4ee0-bba7-4e7484f385ec-kube-api-access-vkdx2\") pod \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") "
Jul 13 19:59:23.724208 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:23.724102    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-tmp-dir\") pod \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") "
Jul 13 19:59:23.724208 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:23.724117    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-model-cache\") pod \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") "
Jul 13 19:59:23.724208 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:23.724146    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-dshm\") pod \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\" (UID: \"a4b80443-c66f-4ee0-bba7-4e7484f385ec\") "
Jul 13 19:59:23.724508 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:23.724384    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-home" (OuterVolumeSpecName: "home") pod "a4b80443-c66f-4ee0-bba7-4e7484f385ec" (UID: "a4b80443-c66f-4ee0-bba7-4e7484f385ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:59:23.724508 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:23.724429    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "a4b80443-c66f-4ee0-bba7-4e7484f385ec" (UID: "a4b80443-c66f-4ee0-bba7-4e7484f385ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:59:23.724508 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:23.724460    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a4b80443-c66f-4ee0-bba7-4e7484f385ec" (UID: "a4b80443-c66f-4ee0-bba7-4e7484f385ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:59:23.726528 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:23.726500    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4b80443-c66f-4ee0-bba7-4e7484f385ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a4b80443-c66f-4ee0-bba7-4e7484f385ec" (UID: "a4b80443-c66f-4ee0-bba7-4e7484f385ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:59:23.726528 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:23.726505    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-dshm" (OuterVolumeSpecName: "dshm") pod "a4b80443-c66f-4ee0-bba7-4e7484f385ec" (UID: "a4b80443-c66f-4ee0-bba7-4e7484f385ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:59:23.726705 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:23.726619    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b80443-c66f-4ee0-bba7-4e7484f385ec-kube-api-access-vkdx2" (OuterVolumeSpecName: "kube-api-access-vkdx2") pod "a4b80443-c66f-4ee0-bba7-4e7484f385ec" (UID: "a4b80443-c66f-4ee0-bba7-4e7484f385ec"). InnerVolumeSpecName "kube-api-access-vkdx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:59:23.824811 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:23.824777    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4b80443-c66f-4ee0-bba7-4e7484f385ec-tls-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:59:23.824811 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:23.824814    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-home\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:59:23.825039 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:23.824832    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkdx2\" (UniqueName: \"kubernetes.io/projected/a4b80443-c66f-4ee0-bba7-4e7484f385ec-kube-api-access-vkdx2\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:59:23.825039 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:23.824865    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-tmp-dir\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:59:23.825039 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:23.824875    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-model-cache\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:59:23.825039 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:23.824882    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4b80443-c66f-4ee0-bba7-4e7484f385ec-dshm\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 19:59:24.160152 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:24.160115    2569 generic.go:356] "Generic (PLEG): container finished" podID="a4b80443-c66f-4ee0-bba7-4e7484f385ec" containerID="9f79933c25aa21d115d4f56b227f78fa84a737ed26df7efae60cf51bebb97e3e" exitCode=0
Jul 13 19:59:24.160349 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:24.160176    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"
Jul 13 19:59:24.160349 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:24.160181    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n" event={"ID":"a4b80443-c66f-4ee0-bba7-4e7484f385ec","Type":"ContainerDied","Data":"9f79933c25aa21d115d4f56b227f78fa84a737ed26df7efae60cf51bebb97e3e"}
Jul 13 19:59:24.160349 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:24.160233    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n" event={"ID":"a4b80443-c66f-4ee0-bba7-4e7484f385ec","Type":"ContainerDied","Data":"be6c4950001d30927132ae1d064670f295eeec63966bf8b8e6e8ff2aff01941c"}
Jul 13 19:59:24.160349 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:24.160262    2569 scope.go:117] "RemoveContainer" containerID="9f79933c25aa21d115d4f56b227f78fa84a737ed26df7efae60cf51bebb97e3e"
Jul 13 19:59:24.170055 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:24.170025    2569 scope.go:117] "RemoveContainer" containerID="9f79933c25aa21d115d4f56b227f78fa84a737ed26df7efae60cf51bebb97e3e"
Jul 13 19:59:24.170347 ip-10-0-139-134 kubenswrapper[2569]: E0713 19:59:24.170325    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f79933c25aa21d115d4f56b227f78fa84a737ed26df7efae60cf51bebb97e3e\": container with ID starting with 9f79933c25aa21d115d4f56b227f78fa84a737ed26df7efae60cf51bebb97e3e not found: ID does not exist" containerID="9f79933c25aa21d115d4f56b227f78fa84a737ed26df7efae60cf51bebb97e3e"
Jul 13 19:59:24.170405 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:24.170355    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f79933c25aa21d115d4f56b227f78fa84a737ed26df7efae60cf51bebb97e3e"} err="failed to get container status \"9f79933c25aa21d115d4f56b227f78fa84a737ed26df7efae60cf51bebb97e3e\": rpc error: code = NotFound desc = could not find container \"9f79933c25aa21d115d4f56b227f78fa84a737ed26df7efae60cf51bebb97e3e\": container with ID starting with 9f79933c25aa21d115d4f56b227f78fa84a737ed26df7efae60cf51bebb97e3e not found: ID does not exist"
Jul 13 19:59:24.186430 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:24.186386    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"]
Jul 13 19:59:24.188487 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:24.188464    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n"]
Jul 13 19:59:25.061568 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:25.061530    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4b80443-c66f-4ee0-bba7-4e7484f385ec" path="/var/lib/kubelet/pods/a4b80443-c66f-4ee0-bba7-4e7484f385ec/volumes"
Jul 13 19:59:39.288838 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:39.288727    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:59:39.298886 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:39.293529    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 19:59:39.298886 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:39.297849    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 19:59:39.302675 ip-10-0-139-134 kubenswrapper[2569]: I0713 19:59:39.302651    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 20:00:10.193776 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.193693    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"]
Jul 13 20:00:10.194248 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.194025    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4b80443-c66f-4ee0-bba7-4e7484f385ec" containerName="main"
Jul 13 20:00:10.194248 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.194036    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4b80443-c66f-4ee0-bba7-4e7484f385ec" containerName="main"
Jul 13 20:00:10.194248 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.194106    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4b80443-c66f-4ee0-bba7-4e7484f385ec" containerName="main"
Jul 13 20:00:10.197070 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.197049    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:10.200067 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.200039    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 13 20:00:10.206820 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.206785    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"]
Jul 13 20:00:10.327655 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.327616    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:10.327878 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.327674    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7btm\" (UniqueName: \"kubernetes.io/projected/600a8ed4-30bb-4ab2-8513-7544de11e2f8-kube-api-access-n7btm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:10.327878 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.327762    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:10.327878 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.327800    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/600a8ed4-30bb-4ab2-8513-7544de11e2f8-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:10.327878 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.327876    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:10.328053 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.327913    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:10.429315 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.429255    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:10.429510 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.429336    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/600a8ed4-30bb-4ab2-8513-7544de11e2f8-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:10.429510 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.429378    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:10.429510 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.429420    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:10.429510 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.429480    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:10.429729 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.429541    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7btm\" (UniqueName: \"kubernetes.io/projected/600a8ed4-30bb-4ab2-8513-7544de11e2f8-kube-api-access-n7btm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:10.429836 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.429811    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:10.429905 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.429840    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:10.429945 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.429927    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:10.432724 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.432692    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:10.432905 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.432749    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/600a8ed4-30bb-4ab2-8513-7544de11e2f8-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:10.442615 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.442575    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7btm\" (UniqueName: \"kubernetes.io/projected/600a8ed4-30bb-4ab2-8513-7544de11e2f8-kube-api-access-n7btm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:10.451241 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.451151    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w"]
Jul 13 20:00:10.454941 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.454912    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w"
Jul 13 20:00:10.457610 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.457582    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-25hgf\""
Jul 13 20:00:10.465144 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.465112    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w"]
Jul 13 20:00:10.508495 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.508455    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:10.630836 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.630801    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6qrj\" (UniqueName: \"kubernetes.io/projected/9a995a06-16c0-4787-8963-18ebd831c98e-kube-api-access-r6qrj\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w\" (UID: \"9a995a06-16c0-4787-8963-18ebd831c98e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w"
Jul 13 20:00:10.630836 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.630843    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a995a06-16c0-4787-8963-18ebd831c98e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w\" (UID: \"9a995a06-16c0-4787-8963-18ebd831c98e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w"
Jul 13 20:00:10.631075 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.630880    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9a995a06-16c0-4787-8963-18ebd831c98e-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w\" (UID: \"9a995a06-16c0-4787-8963-18ebd831c98e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w"
Jul 13 20:00:10.654653 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.654553    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"]
Jul 13 20:00:10.657075 ip-10-0-139-134 kubenswrapper[2569]: W0713 20:00:10.657040    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600a8ed4_30bb_4ab2_8513_7544de11e2f8.slice/crio-86100d54586e36acfbea094496fd3cfb96f36832f397f615fa751938923c281f WatchSource:0}: Error finding container 86100d54586e36acfbea094496fd3cfb96f36832f397f615fa751938923c281f: Status 404 returned error can't find the container with id 86100d54586e36acfbea094496fd3cfb96f36832f397f615fa751938923c281f
Jul 13 20:00:10.731689 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.731649    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6qrj\" (UniqueName: \"kubernetes.io/projected/9a995a06-16c0-4787-8963-18ebd831c98e-kube-api-access-r6qrj\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w\" (UID: \"9a995a06-16c0-4787-8963-18ebd831c98e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w"
Jul 13 20:00:10.731689 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.731694    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a995a06-16c0-4787-8963-18ebd831c98e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w\" (UID: \"9a995a06-16c0-4787-8963-18ebd831c98e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w"
Jul 13 20:00:10.731947 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.731732    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9a995a06-16c0-4787-8963-18ebd831c98e-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w\" (UID: \"9a995a06-16c0-4787-8963-18ebd831c98e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w"
Jul 13 20:00:10.732122 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.732103    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9a995a06-16c0-4787-8963-18ebd831c98e-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w\" (UID: \"9a995a06-16c0-4787-8963-18ebd831c98e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w"
Jul 13 20:00:10.734479 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.734452    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a995a06-16c0-4787-8963-18ebd831c98e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w\" (UID: \"9a995a06-16c0-4787-8963-18ebd831c98e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w"
Jul 13 20:00:10.741141 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.741101    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6qrj\" (UniqueName: \"kubernetes.io/projected/9a995a06-16c0-4787-8963-18ebd831c98e-kube-api-access-r6qrj\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w\" (UID: \"9a995a06-16c0-4787-8963-18ebd831c98e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w"
Jul 13 20:00:10.767055 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.767020    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w"
Jul 13 20:00:10.920091 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:10.920025    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w"]
Jul 13 20:00:10.923068 ip-10-0-139-134 kubenswrapper[2569]: W0713 20:00:10.923039    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a995a06_16c0_4787_8963_18ebd831c98e.slice/crio-40e4bdccaa0dd763471a6e8dd93d38e465ac96faa64b7f3cfbdf18ab77de71b6 WatchSource:0}: Error finding container 40e4bdccaa0dd763471a6e8dd93d38e465ac96faa64b7f3cfbdf18ab77de71b6: Status 404 returned error can't find the container with id 40e4bdccaa0dd763471a6e8dd93d38e465ac96faa64b7f3cfbdf18ab77de71b6
Jul 13 20:00:11.382966 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:11.382925    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w" event={"ID":"9a995a06-16c0-4787-8963-18ebd831c98e","Type":"ContainerStarted","Data":"fa619d4085833715a7b195d8dc88498c9c739ba5590ad7510a6ba2c7890475e7"}
Jul 13 20:00:11.383506 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:11.382985    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w" event={"ID":"9a995a06-16c0-4787-8963-18ebd831c98e","Type":"ContainerStarted","Data":"40e4bdccaa0dd763471a6e8dd93d38e465ac96faa64b7f3cfbdf18ab77de71b6"}
Jul 13 20:00:11.383506 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:11.383040    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w"
Jul 13 20:00:11.384618 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:11.384589    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5" event={"ID":"600a8ed4-30bb-4ab2-8513-7544de11e2f8","Type":"ContainerStarted","Data":"8e6c1d70f7bfe5947f11cb23898b4611711decec4d1514f3a09b45120180c953"}
Jul 13 20:00:11.384758 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:11.384650    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5" event={"ID":"600a8ed4-30bb-4ab2-8513-7544de11e2f8","Type":"ContainerStarted","Data":"86100d54586e36acfbea094496fd3cfb96f36832f397f615fa751938923c281f"}
Jul 13 20:00:11.402237 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:11.402179    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w" podStartSLOduration=1.402161534 podStartE2EDuration="1.402161534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:00:11.401048428 +0000 UTC m=+4232.914847595" watchObservedRunningTime="2026-07-13 20:00:11.402161534 +0000 UTC m=+4232.915960735"
Jul 13 20:00:11.420193 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:11.420136    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5" podStartSLOduration=1.420118088 podStartE2EDuration="1.420118088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:00:11.418613613 +0000 UTC m=+4232.932412779" watchObservedRunningTime="2026-07-13 20:00:11.420118088 +0000 UTC m=+4232.933917253"
Jul 13 20:00:11.509179 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:11.509137    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:20.509483 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:20.509432    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:20.522310 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:20.522257    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:21.443149 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:21.443119    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:00:42.390979 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:42.390942    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w" podUID="9a995a06-16c0-4787-8963-18ebd831c98e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 13 20:00:52.391331 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:00:52.391269    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w" podUID="9a995a06-16c0-4787-8963-18ebd831c98e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 13 20:01:02.391179 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:02.391135    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w" podUID="9a995a06-16c0-4787-8963-18ebd831c98e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 13 20:01:12.391401 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:12.391353    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w" podUID="9a995a06-16c0-4787-8963-18ebd831c98e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 13 20:01:20.056415 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:20.056373    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w" podUID="9a995a06-16c0-4787-8963-18ebd831c98e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 13 20:01:30.057032 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:30.056948    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w" podUID="9a995a06-16c0-4787-8963-18ebd831c98e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 13 20:01:40.057312 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:40.057258    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w" podUID="9a995a06-16c0-4787-8963-18ebd831c98e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 13 20:01:49.646741 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:49.646702    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w"]
Jul 13 20:01:49.649972 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:49.649928    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w" podUID="9a995a06-16c0-4787-8963-18ebd831c98e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 13 20:01:49.651140 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:49.651117    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"]
Jul 13 20:01:49.663412 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:01:49.663362    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 20:01:49.663412 ip-10-0-139-134 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:01:49.663412 ip-10-0-139-134 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 13 20:01:49.663412 ip-10-0-139-134 kubenswrapper[2569]:  > containerID="fa619d4085833715a7b195d8dc88498c9c739ba5590ad7510a6ba2c7890475e7" cmd=["/bin/sleep","15"]
Jul 13 20:01:49.663412 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:49.663433    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w" message="" err=<
Jul 13 20:01:49.663412 ip-10-0-139-134 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:01:49.663412 ip-10-0-139-134 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 13 20:01:49.663412 ip-10-0-139-134 kubenswrapper[2569]:  >
Jul 13 20:01:49.663981 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:01:49.663451    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 20:01:49.663981 ip-10-0-139-134 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:01:49.663981 ip-10-0-139-134 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 13 20:01:49.663981 ip-10-0-139-134 kubenswrapper[2569]:  > pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w" podUID="9a995a06-16c0-4787-8963-18ebd831c98e" containerName="main" containerID="cri-o://fa619d4085833715a7b195d8dc88498c9c739ba5590ad7510a6ba2c7890475e7"
Jul 13 20:01:49.663981 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:49.663504    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w" podUID="9a995a06-16c0-4787-8963-18ebd831c98e" containerName="main" containerID="cri-o://fa619d4085833715a7b195d8dc88498c9c739ba5590ad7510a6ba2c7890475e7" gracePeriod=60
Jul 13 20:01:49.859218 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:49.859138    2569 generic.go:356] "Generic (PLEG): container finished" podID="9a995a06-16c0-4787-8963-18ebd831c98e" containerID="fa619d4085833715a7b195d8dc88498c9c739ba5590ad7510a6ba2c7890475e7" exitCode=0
Jul 13 20:01:49.859428 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:49.859245    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w" event={"ID":"9a995a06-16c0-4787-8963-18ebd831c98e","Type":"ContainerDied","Data":"fa619d4085833715a7b195d8dc88498c9c739ba5590ad7510a6ba2c7890475e7"}
Jul 13 20:01:49.931922 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:49.931891    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w"
Jul 13 20:01:49.940520 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:49.940492    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6qrj\" (UniqueName: \"kubernetes.io/projected/9a995a06-16c0-4787-8963-18ebd831c98e-kube-api-access-r6qrj\") pod \"9a995a06-16c0-4787-8963-18ebd831c98e\" (UID: \"9a995a06-16c0-4787-8963-18ebd831c98e\") "
Jul 13 20:01:49.940664 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:49.940543    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9a995a06-16c0-4787-8963-18ebd831c98e-tokenizer-uds\") pod \"9a995a06-16c0-4787-8963-18ebd831c98e\" (UID: \"9a995a06-16c0-4787-8963-18ebd831c98e\") "
Jul 13 20:01:49.940664 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:49.940604    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a995a06-16c0-4787-8963-18ebd831c98e-tls-certs\") pod \"9a995a06-16c0-4787-8963-18ebd831c98e\" (UID: \"9a995a06-16c0-4787-8963-18ebd831c98e\") "
Jul 13 20:01:49.940869 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:49.940846    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a995a06-16c0-4787-8963-18ebd831c98e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "9a995a06-16c0-4787-8963-18ebd831c98e" (UID: "9a995a06-16c0-4787-8963-18ebd831c98e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:01:49.942783 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:49.942749    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a995a06-16c0-4787-8963-18ebd831c98e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9a995a06-16c0-4787-8963-18ebd831c98e" (UID: "9a995a06-16c0-4787-8963-18ebd831c98e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:01:49.942783 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:49.942770    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a995a06-16c0-4787-8963-18ebd831c98e-kube-api-access-r6qrj" (OuterVolumeSpecName: "kube-api-access-r6qrj") pod "9a995a06-16c0-4787-8963-18ebd831c98e" (UID: "9a995a06-16c0-4787-8963-18ebd831c98e"). InnerVolumeSpecName "kube-api-access-r6qrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:01:50.041422 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:50.041383    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6qrj\" (UniqueName: \"kubernetes.io/projected/9a995a06-16c0-4787-8963-18ebd831c98e-kube-api-access-r6qrj\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:01:50.041422 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:50.041416    2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9a995a06-16c0-4787-8963-18ebd831c98e-tokenizer-uds\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:01:50.041422 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:50.041426    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a995a06-16c0-4787-8963-18ebd831c98e-tls-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:01:50.864716 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:50.864687    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w"
Jul 13 20:01:50.865137 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:50.864692    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w" event={"ID":"9a995a06-16c0-4787-8963-18ebd831c98e","Type":"ContainerDied","Data":"40e4bdccaa0dd763471a6e8dd93d38e465ac96faa64b7f3cfbdf18ab77de71b6"}
Jul 13 20:01:50.865137 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:50.864843    2569 scope.go:117] "RemoveContainer" containerID="fa619d4085833715a7b195d8dc88498c9c739ba5590ad7510a6ba2c7890475e7"
Jul 13 20:01:50.886492 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:50.886451    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w"]
Jul 13 20:01:50.892188 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:50.892156    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w"]
Jul 13 20:01:51.061176 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:01:51.061138    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a995a06-16c0-4787-8963-18ebd831c98e" path="/var/lib/kubelet/pods/9a995a06-16c0-4787-8963-18ebd831c98e/volumes"
Jul 13 20:02:04.677947 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:04.677887    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5" podUID="600a8ed4-30bb-4ab2-8513-7544de11e2f8" containerName="main" containerID="cri-o://8e6c1d70f7bfe5947f11cb23898b4611711decec4d1514f3a09b45120180c953" gracePeriod=45
Jul 13 20:02:04.937992 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:04.937956    2569 generic.go:356] "Generic (PLEG): container finished" podID="600a8ed4-30bb-4ab2-8513-7544de11e2f8" containerID="8e6c1d70f7bfe5947f11cb23898b4611711decec4d1514f3a09b45120180c953" exitCode=0
Jul 13 20:02:04.938154 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:04.938022    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5" event={"ID":"600a8ed4-30bb-4ab2-8513-7544de11e2f8","Type":"ContainerDied","Data":"8e6c1d70f7bfe5947f11cb23898b4611711decec4d1514f3a09b45120180c953"}
Jul 13 20:02:04.938154 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:04.938061    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5" event={"ID":"600a8ed4-30bb-4ab2-8513-7544de11e2f8","Type":"ContainerDied","Data":"86100d54586e36acfbea094496fd3cfb96f36832f397f615fa751938923c281f"}
Jul 13 20:02:04.938154 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:04.938076    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86100d54586e36acfbea094496fd3cfb96f36832f397f615fa751938923c281f"
Jul 13 20:02:04.944737 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:04.944709    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:02:04.975200 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:04.975163    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-home\") pod \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") "
Jul 13 20:02:04.975416 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:04.975216    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/600a8ed4-30bb-4ab2-8513-7544de11e2f8-tls-certs\") pod \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") "
Jul 13 20:02:04.975416 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:04.975245    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-dshm\") pod \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") "
Jul 13 20:02:04.975416 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:04.975271    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-tmp-dir\") pod \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") "
Jul 13 20:02:04.975416 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:04.975335    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-model-cache\") pod \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") "
Jul 13 20:02:04.975416 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:04.975370    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7btm\" (UniqueName: \"kubernetes.io/projected/600a8ed4-30bb-4ab2-8513-7544de11e2f8-kube-api-access-n7btm\") pod \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\" (UID: \"600a8ed4-30bb-4ab2-8513-7544de11e2f8\") "
Jul 13 20:02:04.975680 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:04.975522    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-home" (OuterVolumeSpecName: "home") pod "600a8ed4-30bb-4ab2-8513-7544de11e2f8" (UID: "600a8ed4-30bb-4ab2-8513-7544de11e2f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:02:04.975680 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:04.975623    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "600a8ed4-30bb-4ab2-8513-7544de11e2f8" (UID: "600a8ed4-30bb-4ab2-8513-7544de11e2f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:02:04.975680 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:04.975635    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-model-cache" (OuterVolumeSpecName: "model-cache") pod "600a8ed4-30bb-4ab2-8513-7544de11e2f8" (UID: "600a8ed4-30bb-4ab2-8513-7544de11e2f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:02:04.975831 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:04.975722    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-home\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:02:04.975831 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:04.975742    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-tmp-dir\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:02:04.975831 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:04.975758    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-model-cache\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:02:04.977508 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:04.977484    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/600a8ed4-30bb-4ab2-8513-7544de11e2f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "600a8ed4-30bb-4ab2-8513-7544de11e2f8" (UID: "600a8ed4-30bb-4ab2-8513-7544de11e2f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:02:04.977612 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:04.977576    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600a8ed4-30bb-4ab2-8513-7544de11e2f8-kube-api-access-n7btm" (OuterVolumeSpecName: "kube-api-access-n7btm") pod "600a8ed4-30bb-4ab2-8513-7544de11e2f8" (UID: "600a8ed4-30bb-4ab2-8513-7544de11e2f8"). InnerVolumeSpecName "kube-api-access-n7btm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:02:04.977875 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:04.977846    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-dshm" (OuterVolumeSpecName: "dshm") pod "600a8ed4-30bb-4ab2-8513-7544de11e2f8" (UID: "600a8ed4-30bb-4ab2-8513-7544de11e2f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:02:05.076922 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:05.076879    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7btm\" (UniqueName: \"kubernetes.io/projected/600a8ed4-30bb-4ab2-8513-7544de11e2f8-kube-api-access-n7btm\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:02:05.076922 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:05.076916    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/600a8ed4-30bb-4ab2-8513-7544de11e2f8-tls-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:02:05.076922 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:05.076926    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/600a8ed4-30bb-4ab2-8513-7544de11e2f8-dshm\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:02:05.942302 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:05.942268    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"
Jul 13 20:02:05.959195 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:05.959163    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"]
Jul 13 20:02:05.968713 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:05.968677    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5"]
Jul 13 20:02:07.064765 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:07.064725    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="600a8ed4-30bb-4ab2-8513-7544de11e2f8" path="/var/lib/kubelet/pods/600a8ed4-30bb-4ab2-8513-7544de11e2f8/volumes"
Jul 13 20:02:23.889841 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:23.889798    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"]
Jul 13 20:02:23.890368 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:23.890174    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="600a8ed4-30bb-4ab2-8513-7544de11e2f8" containerName="main"
Jul 13 20:02:23.890368 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:23.890187    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="600a8ed4-30bb-4ab2-8513-7544de11e2f8" containerName="main"
Jul 13 20:02:23.890368 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:23.890196    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a995a06-16c0-4787-8963-18ebd831c98e" containerName="main"
Jul 13 20:02:23.890368 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:23.890202    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a995a06-16c0-4787-8963-18ebd831c98e" containerName="main"
Jul 13 20:02:23.890368 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:23.890253    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="600a8ed4-30bb-4ab2-8513-7544de11e2f8" containerName="main"
Jul 13 20:02:23.890368 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:23.890264    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a995a06-16c0-4787-8963-18ebd831c98e" containerName="main"
Jul 13 20:02:23.894810 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:23.894788    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:23.897892 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:23.897867    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 13 20:02:23.903058 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:23.903030    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"]
Jul 13 20:02:24.036938 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:24.036897    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5732fdb9-7753-47a4-bae9-b8a58b774ac2-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-28w56\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:24.036938 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:24.036935    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxzh9\" (UniqueName: \"kubernetes.io/projected/5732fdb9-7753-47a4-bae9-b8a58b774ac2-kube-api-access-jxzh9\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-28w56\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:24.037213 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:24.036993    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-28w56\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:24.037213 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:24.037018    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-28w56\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:24.037213 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:24.037043    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-28w56\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:24.037213 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:24.037130    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-28w56\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:24.137715 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:24.137667    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-28w56\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:24.137715 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:24.137724    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-28w56\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:24.137983 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:24.137764    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5732fdb9-7753-47a4-bae9-b8a58b774ac2-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-28w56\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:24.137983 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:24.137782    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxzh9\" (UniqueName: \"kubernetes.io/projected/5732fdb9-7753-47a4-bae9-b8a58b774ac2-kube-api-access-jxzh9\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-28w56\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:24.137983 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:24.137816    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-28w56\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:24.137983 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:24.137841    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-28w56\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:24.138183 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:24.138153    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-28w56\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:24.138228 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:24.138180    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-28w56\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:24.138266 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:24.138220    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-28w56\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:24.140373 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:24.140284    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-28w56\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:24.140483 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:24.140461    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5732fdb9-7753-47a4-bae9-b8a58b774ac2-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-28w56\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:24.152980 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:24.152952    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxzh9\" (UniqueName: \"kubernetes.io/projected/5732fdb9-7753-47a4-bae9-b8a58b774ac2-kube-api-access-jxzh9\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-28w56\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:24.205903 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:24.205862    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:24.351400 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:24.351363    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"]
Jul 13 20:02:24.353318 ip-10-0-139-134 kubenswrapper[2569]: W0713 20:02:24.353266    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5732fdb9_7753_47a4_bae9_b8a58b774ac2.slice/crio-26b456a64a86103720ecf24c04da75c34bc2971aaa4b3d287669ce6b9b47ee4b WatchSource:0}: Error finding container 26b456a64a86103720ecf24c04da75c34bc2971aaa4b3d287669ce6b9b47ee4b: Status 404 returned error can't find the container with id 26b456a64a86103720ecf24c04da75c34bc2971aaa4b3d287669ce6b9b47ee4b
Jul 13 20:02:25.033630 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:25.033590    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56" event={"ID":"5732fdb9-7753-47a4-bae9-b8a58b774ac2","Type":"ContainerStarted","Data":"e7e35b714210cfe832a071f768e45aefa54307c5187b443021831372e796377d"}
Jul 13 20:02:25.034025 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:25.033676    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56" event={"ID":"5732fdb9-7753-47a4-bae9-b8a58b774ac2","Type":"ContainerStarted","Data":"26b456a64a86103720ecf24c04da75c34bc2971aaa4b3d287669ce6b9b47ee4b"}
Jul 13 20:02:25.052948 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:25.052884    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56" podStartSLOduration=2.052861696 podStartE2EDuration="2.052861696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:02:25.052231545 +0000 UTC m=+4366.566030711" watchObservedRunningTime="2026-07-13 20:02:25.052861696 +0000 UTC m=+4366.566660863"
Jul 13 20:02:25.206036 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:25.205995    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:34.206991 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:34.206945    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:34.220917 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:34.220885    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:35.092453 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:35.092416    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:02:58.296239 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:02:58.296208    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2_2ce120d6-0e30-45b7-90fc-f7f76654786e/main/0.log"
Jul 13 20:04:32.104894 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:32.104808    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"]
Jul 13 20:04:39.318439 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:39.318330    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 20:04:39.323160 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:39.323137    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 20:04:39.333466 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:39.333439    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 20:04:39.338417 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:39.338387    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 20:04:47.128197 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.128150    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56" podUID="5732fdb9-7753-47a4-bae9-b8a58b774ac2" containerName="main" containerID="cri-o://e7e35b714210cfe832a071f768e45aefa54307c5187b443021831372e796377d" gracePeriod=45
Jul 13 20:04:47.383284 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.383210    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:04:47.430594 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.430540    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-tmp-dir\") pod \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") "
Jul 13 20:04:47.430594 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.430602    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-dshm\") pod \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") "
Jul 13 20:04:47.430855 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.430714    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5732fdb9-7753-47a4-bae9-b8a58b774ac2-tls-certs\") pod \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") "
Jul 13 20:04:47.430855 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.430742    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-model-cache\") pod \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") "
Jul 13 20:04:47.430855 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.430771    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-home\") pod \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") "
Jul 13 20:04:47.430855 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.430807    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxzh9\" (UniqueName: \"kubernetes.io/projected/5732fdb9-7753-47a4-bae9-b8a58b774ac2-kube-api-access-jxzh9\") pod \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\" (UID: \"5732fdb9-7753-47a4-bae9-b8a58b774ac2\") "
Jul 13 20:04:47.431044 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.430882    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5732fdb9-7753-47a4-bae9-b8a58b774ac2" (UID: "5732fdb9-7753-47a4-bae9-b8a58b774ac2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:04:47.431104 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.431069    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-model-cache" (OuterVolumeSpecName: "model-cache") pod "5732fdb9-7753-47a4-bae9-b8a58b774ac2" (UID: "5732fdb9-7753-47a4-bae9-b8a58b774ac2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:04:47.431104 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.431080    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-tmp-dir\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:04:47.431254 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.431219    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-home" (OuterVolumeSpecName: "home") pod "5732fdb9-7753-47a4-bae9-b8a58b774ac2" (UID: "5732fdb9-7753-47a4-bae9-b8a58b774ac2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:04:47.433185 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.433156    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-dshm" (OuterVolumeSpecName: "dshm") pod "5732fdb9-7753-47a4-bae9-b8a58b774ac2" (UID: "5732fdb9-7753-47a4-bae9-b8a58b774ac2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:04:47.433644 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.433607    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5732fdb9-7753-47a4-bae9-b8a58b774ac2-kube-api-access-jxzh9" (OuterVolumeSpecName: "kube-api-access-jxzh9") pod "5732fdb9-7753-47a4-bae9-b8a58b774ac2" (UID: "5732fdb9-7753-47a4-bae9-b8a58b774ac2"). InnerVolumeSpecName "kube-api-access-jxzh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:04:47.433644 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.433613    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5732fdb9-7753-47a4-bae9-b8a58b774ac2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5732fdb9-7753-47a4-bae9-b8a58b774ac2" (UID: "5732fdb9-7753-47a4-bae9-b8a58b774ac2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:04:47.532471 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.532422    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-dshm\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:04:47.532471 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.532461    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5732fdb9-7753-47a4-bae9-b8a58b774ac2-tls-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:04:47.532471 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.532471    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-model-cache\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:04:47.532471 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.532480    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5732fdb9-7753-47a4-bae9-b8a58b774ac2-home\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:04:47.532845 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.532488    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxzh9\" (UniqueName: \"kubernetes.io/projected/5732fdb9-7753-47a4-bae9-b8a58b774ac2-kube-api-access-jxzh9\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:04:47.706990 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.706896    2569 generic.go:356] "Generic (PLEG): container finished" podID="5732fdb9-7753-47a4-bae9-b8a58b774ac2" containerID="e7e35b714210cfe832a071f768e45aefa54307c5187b443021831372e796377d" exitCode=0
Jul 13 20:04:47.706990 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.706938    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"
Jul 13 20:04:47.706990 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.706979    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56" event={"ID":"5732fdb9-7753-47a4-bae9-b8a58b774ac2","Type":"ContainerDied","Data":"e7e35b714210cfe832a071f768e45aefa54307c5187b443021831372e796377d"}
Jul 13 20:04:47.707237 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.707033    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56" event={"ID":"5732fdb9-7753-47a4-bae9-b8a58b774ac2","Type":"ContainerDied","Data":"26b456a64a86103720ecf24c04da75c34bc2971aaa4b3d287669ce6b9b47ee4b"}
Jul 13 20:04:47.707237 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.707055    2569 scope.go:117] "RemoveContainer" containerID="e7e35b714210cfe832a071f768e45aefa54307c5187b443021831372e796377d"
Jul 13 20:04:47.716791 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.716763    2569 scope.go:117] "RemoveContainer" containerID="e7e35b714210cfe832a071f768e45aefa54307c5187b443021831372e796377d"
Jul 13 20:04:47.717136 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:04:47.717105    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7e35b714210cfe832a071f768e45aefa54307c5187b443021831372e796377d\": container with ID starting with e7e35b714210cfe832a071f768e45aefa54307c5187b443021831372e796377d not found: ID does not exist" containerID="e7e35b714210cfe832a071f768e45aefa54307c5187b443021831372e796377d"
Jul 13 20:04:47.717232 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.717147    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7e35b714210cfe832a071f768e45aefa54307c5187b443021831372e796377d"} err="failed to get container status \"e7e35b714210cfe832a071f768e45aefa54307c5187b443021831372e796377d\": rpc error: code = NotFound desc = could not find container \"e7e35b714210cfe832a071f768e45aefa54307c5187b443021831372e796377d\": container with ID starting with e7e35b714210cfe832a071f768e45aefa54307c5187b443021831372e796377d not found: ID does not exist"
Jul 13 20:04:47.728392 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.728359    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"]
Jul 13 20:04:47.731952 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:47.731921    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56"]
Jul 13 20:04:49.061702 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:04:49.061657    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5732fdb9-7753-47a4-bae9-b8a58b774ac2" path="/var/lib/kubelet/pods/5732fdb9-7753-47a4-bae9-b8a58b774ac2/volumes"
Jul 13 20:05:40.007344 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.007303    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"]
Jul 13 20:05:40.007940 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.007917    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5732fdb9-7753-47a4-bae9-b8a58b774ac2" containerName="main"
Jul 13 20:05:40.008027 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.007943    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5732fdb9-7753-47a4-bae9-b8a58b774ac2" containerName="main"
Jul 13 20:05:40.008082 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.008065    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5732fdb9-7753-47a4-bae9-b8a58b774ac2" containerName="main"
Jul 13 20:05:40.012539 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.012446    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:40.022480 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.022458    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 13 20:05:40.022700 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.022679    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-nhkkm\""
Jul 13 20:05:40.044865 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.044830    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"]
Jul 13 20:05:40.054699 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.054664    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"]
Jul 13 20:05:40.058655 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.058632    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:40.096282 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.096253    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"]
Jul 13 20:05:40.104919 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.104881    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bf985a-e88c-4ef4-a049-ca3c2a38028e-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:40.105108 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.104975    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:40.105108 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.105079    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:40.105208 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.105139    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:40.105208 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.105161    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4gnt\" (UniqueName: \"kubernetes.io/projected/67bf985a-e88c-4ef4-a049-ca3c2a38028e-kube-api-access-m4gnt\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:40.105331 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.105214    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:40.205834 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.205793    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:40.206041 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.205852    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:40.206041 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.205923    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:40.206041 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.205970    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:40.206041 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.206019    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:40.206261 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.206044    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:40.206261 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.206070    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4gnt\" (UniqueName: \"kubernetes.io/projected/67bf985a-e88c-4ef4-a049-ca3c2a38028e-kube-api-access-m4gnt\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:40.206261 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.206117    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:40.206261 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.206172    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:40.206261 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.206218    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:40.206261 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.206245    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l6x7\" (UniqueName: \"kubernetes.io/projected/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-kube-api-access-8l6x7\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:40.206593 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.206335    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bf985a-e88c-4ef4-a049-ca3c2a38028e-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:40.206593 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.206407    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:40.206593 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.206434    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:40.206593 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.206488    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:40.208364 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.208341    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:40.208784 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.208767    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bf985a-e88c-4ef4-a049-ca3c2a38028e-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:40.214751 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.214729    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4gnt\" (UniqueName: \"kubernetes.io/projected/67bf985a-e88c-4ef4-a049-ca3c2a38028e-kube-api-access-m4gnt\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:40.307904 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.307799    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:40.307904 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.307850    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:40.308137 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.307917    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:40.308137 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.307950    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:40.308137 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.307976    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l6x7\" (UniqueName: \"kubernetes.io/projected/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-kube-api-access-8l6x7\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:40.308137 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.308029    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:40.308397 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.308346    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:40.308457 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.308420    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:40.308457 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.308445    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:40.310308 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.310273    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:40.310580 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.310563    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:40.318374 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.318349    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l6x7\" (UniqueName: \"kubernetes.io/projected/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-kube-api-access-8l6x7\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:40.324220 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.324197    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:40.369418 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.369382    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:40.409932 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.409846    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2"]
Jul 13 20:05:40.414022 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.413985    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2"
Jul 13 20:05:40.417707 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.417074    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-h7cr7\""
Jul 13 20:05:40.423953 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.423876    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2"]
Jul 13 20:05:40.486508 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.486472    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"]
Jul 13 20:05:40.488566 ip-10-0-139-134 kubenswrapper[2569]: W0713 20:05:40.488538    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67bf985a_e88c_4ef4_a049_ca3c2a38028e.slice/crio-685a13f1dabd208b86f67e1d1593d87cd608a274708383ce718f361e4db8801b WatchSource:0}: Error finding container 685a13f1dabd208b86f67e1d1593d87cd608a274708383ce718f361e4db8801b: Status 404 returned error can't find the container with id 685a13f1dabd208b86f67e1d1593d87cd608a274708383ce718f361e4db8801b
Jul 13 20:05:40.490974 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.490953    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 20:05:40.510456 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.510424    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brnnz\" (UniqueName: \"kubernetes.io/projected/bcc0bc37-fe82-4845-9750-c19b6e6fff20-kube-api-access-brnnz\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2\" (UID: \"bcc0bc37-fe82-4845-9750-c19b6e6fff20\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2"
Jul 13 20:05:40.510626 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.510503    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcc0bc37-fe82-4845-9750-c19b6e6fff20-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2\" (UID: \"bcc0bc37-fe82-4845-9750-c19b6e6fff20\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2"
Jul 13 20:05:40.510626 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.510539    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bcc0bc37-fe82-4845-9750-c19b6e6fff20-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2\" (UID: \"bcc0bc37-fe82-4845-9750-c19b6e6fff20\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2"
Jul 13 20:05:40.535964 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.535939    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"]
Jul 13 20:05:40.538224 ip-10-0-139-134 kubenswrapper[2569]: W0713 20:05:40.538198    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e53764_1d6a_4f60_b8cf_c96c98ba79aa.slice/crio-ad13a6eea36b43038cdb3a33f4da8b1794dc3d046720b8c7a8a7561ca8955ab7 WatchSource:0}: Error finding container ad13a6eea36b43038cdb3a33f4da8b1794dc3d046720b8c7a8a7561ca8955ab7: Status 404 returned error can't find the container with id ad13a6eea36b43038cdb3a33f4da8b1794dc3d046720b8c7a8a7561ca8955ab7
Jul 13 20:05:40.611116 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.611068    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcc0bc37-fe82-4845-9750-c19b6e6fff20-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2\" (UID: \"bcc0bc37-fe82-4845-9750-c19b6e6fff20\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2"
Jul 13 20:05:40.611279 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.611135    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bcc0bc37-fe82-4845-9750-c19b6e6fff20-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2\" (UID: \"bcc0bc37-fe82-4845-9750-c19b6e6fff20\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2"
Jul 13 20:05:40.611279 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.611217    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brnnz\" (UniqueName: \"kubernetes.io/projected/bcc0bc37-fe82-4845-9750-c19b6e6fff20-kube-api-access-brnnz\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2\" (UID: \"bcc0bc37-fe82-4845-9750-c19b6e6fff20\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2"
Jul 13 20:05:40.611611 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.611590    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bcc0bc37-fe82-4845-9750-c19b6e6fff20-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2\" (UID: \"bcc0bc37-fe82-4845-9750-c19b6e6fff20\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2"
Jul 13 20:05:40.613759 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.613738    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcc0bc37-fe82-4845-9750-c19b6e6fff20-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2\" (UID: \"bcc0bc37-fe82-4845-9750-c19b6e6fff20\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2"
Jul 13 20:05:40.619200 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.619175    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brnnz\" (UniqueName: \"kubernetes.io/projected/bcc0bc37-fe82-4845-9750-c19b6e6fff20-kube-api-access-brnnz\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2\" (UID: \"bcc0bc37-fe82-4845-9750-c19b6e6fff20\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2"
Jul 13 20:05:40.729978 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.729943    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2"
Jul 13 20:05:40.884829 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.884781    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2"]
Jul 13 20:05:40.887413 ip-10-0-139-134 kubenswrapper[2569]: W0713 20:05:40.887373    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc0bc37_fe82_4845_9750_c19b6e6fff20.slice/crio-8d284c1470a565068ade43610428789cb58336ace1f30940c96ce32917492a95 WatchSource:0}: Error finding container 8d284c1470a565068ade43610428789cb58336ace1f30940c96ce32917492a95: Status 404 returned error can't find the container with id 8d284c1470a565068ade43610428789cb58336ace1f30940c96ce32917492a95
Jul 13 20:05:40.970164 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.970130    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k" event={"ID":"67bf985a-e88c-4ef4-a049-ca3c2a38028e","Type":"ContainerStarted","Data":"685a13f1dabd208b86f67e1d1593d87cd608a274708383ce718f361e4db8801b"}
Jul 13 20:05:40.972312 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.972238    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx" event={"ID":"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa","Type":"ContainerStarted","Data":"fe97fe2afe53d696fb57499e81acae36f805322a9abdbc0e4391674f27ead154"}
Jul 13 20:05:40.972446 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.972327    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx" event={"ID":"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa","Type":"ContainerStarted","Data":"ad13a6eea36b43038cdb3a33f4da8b1794dc3d046720b8c7a8a7561ca8955ab7"}
Jul 13 20:05:40.973948 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.973924    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2" event={"ID":"bcc0bc37-fe82-4845-9750-c19b6e6fff20","Type":"ContainerStarted","Data":"8d284c1470a565068ade43610428789cb58336ace1f30940c96ce32917492a95"}
Jul 13 20:05:40.974063 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.974050    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2"
Jul 13 20:05:40.990926 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:40.990847    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx" podStartSLOduration=0.990827441 podStartE2EDuration="990.827441ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:05:40.989140764 +0000 UTC m=+4562.502939942" watchObservedRunningTime="2026-07-13 20:05:40.990827441 +0000 UTC m=+4562.504626608"
Jul 13 20:05:41.006721 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:41.006661    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2" podStartSLOduration=1.006644477 podStartE2EDuration="1.006644477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:05:41.004829035 +0000 UTC m=+4562.518628202" watchObservedRunningTime="2026-07-13 20:05:41.006644477 +0000 UTC m=+4562.520443645"
Jul 13 20:05:41.370045 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:41.370019    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:41.981019 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:41.980984    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k" event={"ID":"67bf985a-e88c-4ef4-a049-ca3c2a38028e","Type":"ContainerStarted","Data":"2bff15567b3a651c3546c4cd6dd0412450b68887f189a376ff2b5b8543ee2b7b"}
Jul 13 20:05:41.981220 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:41.981047    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:41.982603 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:41.982571    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2" event={"ID":"bcc0bc37-fe82-4845-9750-c19b6e6fff20","Type":"ContainerStarted","Data":"4197cfda83c17a898f8178e69b382e3077772a72c3c17a1eb719c9178ca3366d"}
Jul 13 20:05:42.990174 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:42.990138    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k" event={"ID":"67bf985a-e88c-4ef4-a049-ca3c2a38028e","Type":"ContainerStarted","Data":"116bf3b7a29ba2d15ac00d1527e03df21d58a1f7c0f298c2f3e9df1790ebebe6"}
Jul 13 20:05:43.009704 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:43.009643    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k" podStartSLOduration=3.17840013 podStartE2EDuration="4.009620998s" totalImagesPullingTime="831.220868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:05:43.008082848 +0000 UTC m=+4564.521882019" watchObservedRunningTime="2026-07-13 20:05:43.009620998 +0000 UTC m=+4564.523420163"
Jul 13 20:05:43.325148 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:43.325055    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:50.325397 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:50.325350    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:50.337963 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:50.337935    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:50.370244 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:50.370199    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:50.385249 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:50.385218    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:05:51.041819 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:51.041790    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:51.042126 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:51.042103    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:05:51.042212 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:05:51.042166    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:06:12.992507 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:12.992424    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2"
Jul 13 20:06:39.524312 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:39.524251    2569 scope.go:117] "RemoveContainer" containerID="8e6c1d70f7bfe5947f11cb23898b4611711decec4d1514f3a09b45120180c953"
Jul 13 20:06:54.865846 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:54.865813    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"]
Jul 13 20:06:54.869260 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:54.869224    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"]
Jul 13 20:06:54.871464 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:54.871435    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2"]
Jul 13 20:06:54.881435 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:06:54.881391    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 20:06:54.881435 ip-10-0-139-134 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:06:54.881435 ip-10-0-139-134 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 13 20:06:54.881435 ip-10-0-139-134 kubenswrapper[2569]:  > containerID="4197cfda83c17a898f8178e69b382e3077772a72c3c17a1eb719c9178ca3366d" cmd=["/bin/sleep","15"]
Jul 13 20:06:54.881676 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:54.881458    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2" message="" err=<
Jul 13 20:06:54.881676 ip-10-0-139-134 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:06:54.881676 ip-10-0-139-134 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 13 20:06:54.881676 ip-10-0-139-134 kubenswrapper[2569]:  >
Jul 13 20:06:54.881676 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:06:54.881477    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 20:06:54.881676 ip-10-0-139-134 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:06:54.881676 ip-10-0-139-134 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 13 20:06:54.881676 ip-10-0-139-134 kubenswrapper[2569]:  > pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2" podUID="bcc0bc37-fe82-4845-9750-c19b6e6fff20" containerName="main" containerID="cri-o://4197cfda83c17a898f8178e69b382e3077772a72c3c17a1eb719c9178ca3366d"
Jul 13 20:06:54.881676 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:54.881535    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2" podUID="bcc0bc37-fe82-4845-9750-c19b6e6fff20" containerName="main" containerID="cri-o://4197cfda83c17a898f8178e69b382e3077772a72c3c17a1eb719c9178ca3366d" gracePeriod=60
Jul 13 20:06:55.139277 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:55.139247    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2"
Jul 13 20:06:55.220389 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:55.220345    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcc0bc37-fe82-4845-9750-c19b6e6fff20-tls-certs\") pod \"bcc0bc37-fe82-4845-9750-c19b6e6fff20\" (UID: \"bcc0bc37-fe82-4845-9750-c19b6e6fff20\") "
Jul 13 20:06:55.220602 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:55.220435    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bcc0bc37-fe82-4845-9750-c19b6e6fff20-tokenizer-uds\") pod \"bcc0bc37-fe82-4845-9750-c19b6e6fff20\" (UID: \"bcc0bc37-fe82-4845-9750-c19b6e6fff20\") "
Jul 13 20:06:55.220602 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:55.220482    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brnnz\" (UniqueName: \"kubernetes.io/projected/bcc0bc37-fe82-4845-9750-c19b6e6fff20-kube-api-access-brnnz\") pod \"bcc0bc37-fe82-4845-9750-c19b6e6fff20\" (UID: \"bcc0bc37-fe82-4845-9750-c19b6e6fff20\") "
Jul 13 20:06:55.220712 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:55.220696    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc0bc37-fe82-4845-9750-c19b6e6fff20-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "bcc0bc37-fe82-4845-9750-c19b6e6fff20" (UID: "bcc0bc37-fe82-4845-9750-c19b6e6fff20"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:06:55.220781 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:55.220766    2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bcc0bc37-fe82-4845-9750-c19b6e6fff20-tokenizer-uds\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:06:55.222671 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:55.222645    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc0bc37-fe82-4845-9750-c19b6e6fff20-kube-api-access-brnnz" (OuterVolumeSpecName: "kube-api-access-brnnz") pod "bcc0bc37-fe82-4845-9750-c19b6e6fff20" (UID: "bcc0bc37-fe82-4845-9750-c19b6e6fff20"). InnerVolumeSpecName "kube-api-access-brnnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:06:55.222754 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:55.222733    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcc0bc37-fe82-4845-9750-c19b6e6fff20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bcc0bc37-fe82-4845-9750-c19b6e6fff20" (UID: "bcc0bc37-fe82-4845-9750-c19b6e6fff20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:06:55.321425 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:55.321384    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brnnz\" (UniqueName: \"kubernetes.io/projected/bcc0bc37-fe82-4845-9750-c19b6e6fff20-kube-api-access-brnnz\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:06:55.321425 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:55.321421    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcc0bc37-fe82-4845-9750-c19b6e6fff20-tls-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:06:55.343307 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:55.343258    2569 generic.go:356] "Generic (PLEG): container finished" podID="bcc0bc37-fe82-4845-9750-c19b6e6fff20" containerID="4197cfda83c17a898f8178e69b382e3077772a72c3c17a1eb719c9178ca3366d" exitCode=0
Jul 13 20:06:55.343505 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:55.343332    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2"
Jul 13 20:06:55.343505 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:55.343350    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2" event={"ID":"bcc0bc37-fe82-4845-9750-c19b6e6fff20","Type":"ContainerDied","Data":"4197cfda83c17a898f8178e69b382e3077772a72c3c17a1eb719c9178ca3366d"}
Jul 13 20:06:55.343505 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:55.343408    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2" event={"ID":"bcc0bc37-fe82-4845-9750-c19b6e6fff20","Type":"ContainerDied","Data":"8d284c1470a565068ade43610428789cb58336ace1f30940c96ce32917492a95"}
Jul 13 20:06:55.343505 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:55.343428    2569 scope.go:117] "RemoveContainer" containerID="4197cfda83c17a898f8178e69b382e3077772a72c3c17a1eb719c9178ca3366d"
Jul 13 20:06:55.353023 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:55.353003    2569 scope.go:117] "RemoveContainer" containerID="4197cfda83c17a898f8178e69b382e3077772a72c3c17a1eb719c9178ca3366d"
Jul 13 20:06:55.353327 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:06:55.353304    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4197cfda83c17a898f8178e69b382e3077772a72c3c17a1eb719c9178ca3366d\": container with ID starting with 4197cfda83c17a898f8178e69b382e3077772a72c3c17a1eb719c9178ca3366d not found: ID does not exist" containerID="4197cfda83c17a898f8178e69b382e3077772a72c3c17a1eb719c9178ca3366d"
Jul 13 20:06:55.353383 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:55.353335    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4197cfda83c17a898f8178e69b382e3077772a72c3c17a1eb719c9178ca3366d"} err="failed to get container status \"4197cfda83c17a898f8178e69b382e3077772a72c3c17a1eb719c9178ca3366d\": rpc error: code = NotFound desc = could not find container \"4197cfda83c17a898f8178e69b382e3077772a72c3c17a1eb719c9178ca3366d\": container with ID starting with 4197cfda83c17a898f8178e69b382e3077772a72c3c17a1eb719c9178ca3366d not found: ID does not exist"
Jul 13 20:06:55.364723 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:55.364685    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2"]
Jul 13 20:06:55.368054 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:55.368025    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59pfml2"]
Jul 13 20:06:57.062273 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:06:57.062239    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcc0bc37-fe82-4845-9750-c19b6e6fff20" path="/var/lib/kubelet/pods/bcc0bc37-fe82-4845-9750-c19b6e6fff20/volumes"
Jul 13 20:07:09.890009 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:09.889949    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k" podUID="67bf985a-e88c-4ef4-a049-ca3c2a38028e" containerName="main" containerID="cri-o://116bf3b7a29ba2d15ac00d1527e03df21d58a1f7c0f298c2f3e9df1790ebebe6" gracePeriod=45
Jul 13 20:07:09.893367 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:09.893328    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx" podUID="c6e53764-1d6a-4f60-b8cf-c96c98ba79aa" containerName="main" containerID="cri-o://fe97fe2afe53d696fb57499e81acae36f805322a9abdbc0e4391674f27ead154" gracePeriod=45
Jul 13 20:07:10.008206 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.008151    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k" podUID="67bf985a-e88c-4ef4-a049-ca3c2a38028e" containerName="llm-d-routing-sidecar" containerID="cri-o://2bff15567b3a651c3546c4cd6dd0412450b68887f189a376ff2b5b8543ee2b7b" gracePeriod=45
Jul 13 20:07:10.030490 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.030444    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx" podUID="c6e53764-1d6a-4f60-b8cf-c96c98ba79aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 13 20:07:10.030615 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.030486    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k" podUID="67bf985a-e88c-4ef4-a049-ca3c2a38028e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Jul 13 20:07:10.162850 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.162819    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:07:10.262386 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.262364    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-tls-certs\") pod \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") "
Jul 13 20:07:10.262532 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.262439    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-home\") pod \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") "
Jul 13 20:07:10.262596 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.262546    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-model-cache\") pod \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") "
Jul 13 20:07:10.262656 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.262597    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8l6x7\" (UniqueName: \"kubernetes.io/projected/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-kube-api-access-8l6x7\") pod \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") "
Jul 13 20:07:10.262712 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.262659    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-dshm\") pod \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") "
Jul 13 20:07:10.262712 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.262667    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-home" (OuterVolumeSpecName: "home") pod "c6e53764-1d6a-4f60-b8cf-c96c98ba79aa" (UID: "c6e53764-1d6a-4f60-b8cf-c96c98ba79aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:07:10.262816 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.262709    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-tmp-dir\") pod \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\" (UID: \"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa\") "
Jul 13 20:07:10.262866 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.262824    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6e53764-1d6a-4f60-b8cf-c96c98ba79aa" (UID: "c6e53764-1d6a-4f60-b8cf-c96c98ba79aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:07:10.263092 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.263042    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6e53764-1d6a-4f60-b8cf-c96c98ba79aa" (UID: "c6e53764-1d6a-4f60-b8cf-c96c98ba79aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:07:10.263092 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.263058    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-model-cache\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:07:10.263092 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.263081    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-home\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:07:10.264826 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.264787    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-dshm" (OuterVolumeSpecName: "dshm") pod "c6e53764-1d6a-4f60-b8cf-c96c98ba79aa" (UID: "c6e53764-1d6a-4f60-b8cf-c96c98ba79aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:07:10.264826 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.264811    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6e53764-1d6a-4f60-b8cf-c96c98ba79aa" (UID: "c6e53764-1d6a-4f60-b8cf-c96c98ba79aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:07:10.264986 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.264905    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-kube-api-access-8l6x7" (OuterVolumeSpecName: "kube-api-access-8l6x7") pod "c6e53764-1d6a-4f60-b8cf-c96c98ba79aa" (UID: "c6e53764-1d6a-4f60-b8cf-c96c98ba79aa"). InnerVolumeSpecName "kube-api-access-8l6x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:07:10.265978 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.265961    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:07:10.364036 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.364002    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-dshm\") pod \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") "
Jul 13 20:07:10.364244 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.364110    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bf985a-e88c-4ef4-a049-ca3c2a38028e-tls-certs\") pod \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") "
Jul 13 20:07:10.364244 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.364135    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-tmp-dir\") pod \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") "
Jul 13 20:07:10.364244 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.364157    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4gnt\" (UniqueName: \"kubernetes.io/projected/67bf985a-e88c-4ef4-a049-ca3c2a38028e-kube-api-access-m4gnt\") pod \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") "
Jul 13 20:07:10.364244 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.364180    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-model-cache\") pod \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") "
Jul 13 20:07:10.364244 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.364204    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-home\") pod \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\" (UID: \"67bf985a-e88c-4ef4-a049-ca3c2a38028e\") "
Jul 13 20:07:10.364530 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.364434    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-dshm\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:07:10.364530 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.364455    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-tmp-dir\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:07:10.364530 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.364469    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-tls-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:07:10.364530 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.364483    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l6x7\" (UniqueName: \"kubernetes.io/projected/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa-kube-api-access-8l6x7\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:07:10.364530 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.364492    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67bf985a-e88c-4ef4-a049-ca3c2a38028e" (UID: "67bf985a-e88c-4ef4-a049-ca3c2a38028e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:07:10.364707 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.364502    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-model-cache" (OuterVolumeSpecName: "model-cache") pod "67bf985a-e88c-4ef4-a049-ca3c2a38028e" (UID: "67bf985a-e88c-4ef4-a049-ca3c2a38028e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:07:10.364707 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.364568    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-home" (OuterVolumeSpecName: "home") pod "67bf985a-e88c-4ef4-a049-ca3c2a38028e" (UID: "67bf985a-e88c-4ef4-a049-ca3c2a38028e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:07:10.366362 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.366340    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-dshm" (OuterVolumeSpecName: "dshm") pod "67bf985a-e88c-4ef4-a049-ca3c2a38028e" (UID: "67bf985a-e88c-4ef4-a049-ca3c2a38028e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:07:10.366428 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.366362    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67bf985a-e88c-4ef4-a049-ca3c2a38028e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67bf985a-e88c-4ef4-a049-ca3c2a38028e" (UID: "67bf985a-e88c-4ef4-a049-ca3c2a38028e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:07:10.366428 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.366380    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67bf985a-e88c-4ef4-a049-ca3c2a38028e-kube-api-access-m4gnt" (OuterVolumeSpecName: "kube-api-access-m4gnt") pod "67bf985a-e88c-4ef4-a049-ca3c2a38028e" (UID: "67bf985a-e88c-4ef4-a049-ca3c2a38028e"). InnerVolumeSpecName "kube-api-access-m4gnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:07:10.417422 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.417321    2569 generic.go:356] "Generic (PLEG): container finished" podID="67bf985a-e88c-4ef4-a049-ca3c2a38028e" containerID="116bf3b7a29ba2d15ac00d1527e03df21d58a1f7c0f298c2f3e9df1790ebebe6" exitCode=0
Jul 13 20:07:10.417422 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.417348    2569 generic.go:356] "Generic (PLEG): container finished" podID="67bf985a-e88c-4ef4-a049-ca3c2a38028e" containerID="2bff15567b3a651c3546c4cd6dd0412450b68887f189a376ff2b5b8543ee2b7b" exitCode=0
Jul 13 20:07:10.417422 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.417380    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"
Jul 13 20:07:10.417779 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.417419    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k" event={"ID":"67bf985a-e88c-4ef4-a049-ca3c2a38028e","Type":"ContainerDied","Data":"116bf3b7a29ba2d15ac00d1527e03df21d58a1f7c0f298c2f3e9df1790ebebe6"}
Jul 13 20:07:10.417779 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.417474    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k" event={"ID":"67bf985a-e88c-4ef4-a049-ca3c2a38028e","Type":"ContainerDied","Data":"2bff15567b3a651c3546c4cd6dd0412450b68887f189a376ff2b5b8543ee2b7b"}
Jul 13 20:07:10.417779 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.417494    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k" event={"ID":"67bf985a-e88c-4ef4-a049-ca3c2a38028e","Type":"ContainerDied","Data":"685a13f1dabd208b86f67e1d1593d87cd608a274708383ce718f361e4db8801b"}
Jul 13 20:07:10.417779 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.417519    2569 scope.go:117] "RemoveContainer" containerID="116bf3b7a29ba2d15ac00d1527e03df21d58a1f7c0f298c2f3e9df1790ebebe6"
Jul 13 20:07:10.418771 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.418743    2569 generic.go:356] "Generic (PLEG): container finished" podID="c6e53764-1d6a-4f60-b8cf-c96c98ba79aa" containerID="fe97fe2afe53d696fb57499e81acae36f805322a9abdbc0e4391674f27ead154" exitCode=0
Jul 13 20:07:10.418856 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.418810    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx" event={"ID":"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa","Type":"ContainerDied","Data":"fe97fe2afe53d696fb57499e81acae36f805322a9abdbc0e4391674f27ead154"}
Jul 13 20:07:10.418856 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.418836    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"
Jul 13 20:07:10.418856 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.418851    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx" event={"ID":"c6e53764-1d6a-4f60-b8cf-c96c98ba79aa","Type":"ContainerDied","Data":"ad13a6eea36b43038cdb3a33f4da8b1794dc3d046720b8c7a8a7561ca8955ab7"}
Jul 13 20:07:10.427239 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.427216    2569 scope.go:117] "RemoveContainer" containerID="2bff15567b3a651c3546c4cd6dd0412450b68887f189a376ff2b5b8543ee2b7b"
Jul 13 20:07:10.435315 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.435281    2569 scope.go:117] "RemoveContainer" containerID="116bf3b7a29ba2d15ac00d1527e03df21d58a1f7c0f298c2f3e9df1790ebebe6"
Jul 13 20:07:10.435583 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:07:10.435566    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"116bf3b7a29ba2d15ac00d1527e03df21d58a1f7c0f298c2f3e9df1790ebebe6\": container with ID starting with 116bf3b7a29ba2d15ac00d1527e03df21d58a1f7c0f298c2f3e9df1790ebebe6 not found: ID does not exist" containerID="116bf3b7a29ba2d15ac00d1527e03df21d58a1f7c0f298c2f3e9df1790ebebe6"
Jul 13 20:07:10.435640 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.435594    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"116bf3b7a29ba2d15ac00d1527e03df21d58a1f7c0f298c2f3e9df1790ebebe6"} err="failed to get container status \"116bf3b7a29ba2d15ac00d1527e03df21d58a1f7c0f298c2f3e9df1790ebebe6\": rpc error: code = NotFound desc = could not find container \"116bf3b7a29ba2d15ac00d1527e03df21d58a1f7c0f298c2f3e9df1790ebebe6\": container with ID starting with 116bf3b7a29ba2d15ac00d1527e03df21d58a1f7c0f298c2f3e9df1790ebebe6 not found: ID does not exist"
Jul 13 20:07:10.435640 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.435613    2569 scope.go:117] "RemoveContainer" containerID="2bff15567b3a651c3546c4cd6dd0412450b68887f189a376ff2b5b8543ee2b7b"
Jul 13 20:07:10.435857 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:07:10.435839    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bff15567b3a651c3546c4cd6dd0412450b68887f189a376ff2b5b8543ee2b7b\": container with ID starting with 2bff15567b3a651c3546c4cd6dd0412450b68887f189a376ff2b5b8543ee2b7b not found: ID does not exist" containerID="2bff15567b3a651c3546c4cd6dd0412450b68887f189a376ff2b5b8543ee2b7b"
Jul 13 20:07:10.435903 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.435867    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bff15567b3a651c3546c4cd6dd0412450b68887f189a376ff2b5b8543ee2b7b"} err="failed to get container status \"2bff15567b3a651c3546c4cd6dd0412450b68887f189a376ff2b5b8543ee2b7b\": rpc error: code = NotFound desc = could not find container \"2bff15567b3a651c3546c4cd6dd0412450b68887f189a376ff2b5b8543ee2b7b\": container with ID starting with 2bff15567b3a651c3546c4cd6dd0412450b68887f189a376ff2b5b8543ee2b7b not found: ID does not exist"
Jul 13 20:07:10.435903 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.435883    2569 scope.go:117] "RemoveContainer" containerID="116bf3b7a29ba2d15ac00d1527e03df21d58a1f7c0f298c2f3e9df1790ebebe6"
Jul 13 20:07:10.436083 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.436063    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"116bf3b7a29ba2d15ac00d1527e03df21d58a1f7c0f298c2f3e9df1790ebebe6"} err="failed to get container status \"116bf3b7a29ba2d15ac00d1527e03df21d58a1f7c0f298c2f3e9df1790ebebe6\": rpc error: code = NotFound desc = could not find container \"116bf3b7a29ba2d15ac00d1527e03df21d58a1f7c0f298c2f3e9df1790ebebe6\": container with ID starting with 116bf3b7a29ba2d15ac00d1527e03df21d58a1f7c0f298c2f3e9df1790ebebe6 not found: ID does not exist"
Jul 13 20:07:10.436155 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.436087    2569 scope.go:117] "RemoveContainer" containerID="2bff15567b3a651c3546c4cd6dd0412450b68887f189a376ff2b5b8543ee2b7b"
Jul 13 20:07:10.436491 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.436471    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bff15567b3a651c3546c4cd6dd0412450b68887f189a376ff2b5b8543ee2b7b"} err="failed to get container status \"2bff15567b3a651c3546c4cd6dd0412450b68887f189a376ff2b5b8543ee2b7b\": rpc error: code = NotFound desc = could not find container \"2bff15567b3a651c3546c4cd6dd0412450b68887f189a376ff2b5b8543ee2b7b\": container with ID starting with 2bff15567b3a651c3546c4cd6dd0412450b68887f189a376ff2b5b8543ee2b7b not found: ID does not exist"
Jul 13 20:07:10.436564 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.436492    2569 scope.go:117] "RemoveContainer" containerID="fe97fe2afe53d696fb57499e81acae36f805322a9abdbc0e4391674f27ead154"
Jul 13 20:07:10.439974 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.439952    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"]
Jul 13 20:07:10.445929 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.445909    2569 scope.go:117] "RemoveContainer" containerID="fe97fe2afe53d696fb57499e81acae36f805322a9abdbc0e4391674f27ead154"
Jul 13 20:07:10.446278 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:07:10.446249    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe97fe2afe53d696fb57499e81acae36f805322a9abdbc0e4391674f27ead154\": container with ID starting with fe97fe2afe53d696fb57499e81acae36f805322a9abdbc0e4391674f27ead154 not found: ID does not exist" containerID="fe97fe2afe53d696fb57499e81acae36f805322a9abdbc0e4391674f27ead154"
Jul 13 20:07:10.446403 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.446275    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe97fe2afe53d696fb57499e81acae36f805322a9abdbc0e4391674f27ead154"} err="failed to get container status \"fe97fe2afe53d696fb57499e81acae36f805322a9abdbc0e4391674f27ead154\": rpc error: code = NotFound desc = could not find container \"fe97fe2afe53d696fb57499e81acae36f805322a9abdbc0e4391674f27ead154\": container with ID starting with fe97fe2afe53d696fb57499e81acae36f805322a9abdbc0e4391674f27ead154 not found: ID does not exist"
Jul 13 20:07:10.446944 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.446924    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-hnm4k"]
Jul 13 20:07:10.458577 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.458548    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"]
Jul 13 20:07:10.462052 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.462029    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-6ppgx"]
Jul 13 20:07:10.465626 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.465607    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bf985a-e88c-4ef4-a049-ca3c2a38028e-tls-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:07:10.465691 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.465630    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-tmp-dir\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:07:10.465691 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.465639    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4gnt\" (UniqueName: \"kubernetes.io/projected/67bf985a-e88c-4ef4-a049-ca3c2a38028e-kube-api-access-m4gnt\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:07:10.465691 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.465650    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-model-cache\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:07:10.465691 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.465660    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-home\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:07:10.465691 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:10.465667    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bf985a-e88c-4ef4-a049-ca3c2a38028e-dshm\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:07:11.060875 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:11.060838    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67bf985a-e88c-4ef4-a049-ca3c2a38028e" path="/var/lib/kubelet/pods/67bf985a-e88c-4ef4-a049-ca3c2a38028e/volumes"
Jul 13 20:07:11.061301 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:07:11.061274    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6e53764-1d6a-4f60-b8cf-c96c98ba79aa" path="/var/lib/kubelet/pods/c6e53764-1d6a-4f60-b8cf-c96c98ba79aa/volumes"
Jul 13 20:08:03.701151 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.701115    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"]
Jul 13 20:08:03.701600 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.701467    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6e53764-1d6a-4f60-b8cf-c96c98ba79aa" containerName="main"
Jul 13 20:08:03.701600 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.701479    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6e53764-1d6a-4f60-b8cf-c96c98ba79aa" containerName="main"
Jul 13 20:08:03.701600 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.701488    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67bf985a-e88c-4ef4-a049-ca3c2a38028e" containerName="main"
Jul 13 20:08:03.701600 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.701492    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="67bf985a-e88c-4ef4-a049-ca3c2a38028e" containerName="main"
Jul 13 20:08:03.701600 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.701503    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcc0bc37-fe82-4845-9750-c19b6e6fff20" containerName="main"
Jul 13 20:08:03.701600 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.701510    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcc0bc37-fe82-4845-9750-c19b6e6fff20" containerName="main"
Jul 13 20:08:03.701600 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.701517    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67bf985a-e88c-4ef4-a049-ca3c2a38028e" containerName="llm-d-routing-sidecar"
Jul 13 20:08:03.701600 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.701522    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="67bf985a-e88c-4ef4-a049-ca3c2a38028e" containerName="llm-d-routing-sidecar"
Jul 13 20:08:03.701600 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.701580    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6e53764-1d6a-4f60-b8cf-c96c98ba79aa" containerName="main"
Jul 13 20:08:03.701600 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.701587    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="67bf985a-e88c-4ef4-a049-ca3c2a38028e" containerName="main"
Jul 13 20:08:03.701600 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.701593    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="67bf985a-e88c-4ef4-a049-ca3c2a38028e" containerName="llm-d-routing-sidecar"
Jul 13 20:08:03.701600 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.701600    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcc0bc37-fe82-4845-9750-c19b6e6fff20" containerName="main"
Jul 13 20:08:03.705012 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.704986    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:03.709595 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.708985    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 13 20:08:03.709595 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.709027    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-nkfpg\""
Jul 13 20:08:03.712630 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.712605    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"]
Jul 13 20:08:03.717145 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.717124    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:03.718474 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.718445    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"]
Jul 13 20:08:03.729515 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.729487    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"]
Jul 13 20:08:03.811083 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.811037    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:03.811342 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.811090    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:03.811342 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.811114    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:03.811342 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.811136    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:03.811342 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.811166    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b635b45b-f504-4cea-be9f-26568921a553-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:03.811342 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.811274    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6x6q\" (UniqueName: \"kubernetes.io/projected/776cf4b4-81f0-4db9-8188-65d6c329f512-kube-api-access-v6x6q\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:03.811342 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.811339    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776cf4b4-81f0-4db9-8188-65d6c329f512-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:03.811643 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.811368    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:03.811643 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.811394    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:03.811643 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.811424    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:03.811643 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.811448    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:03.811643 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.811483    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wznfs\" (UniqueName: \"kubernetes.io/projected/b635b45b-f504-4cea-be9f-26568921a553-kube-api-access-wznfs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:03.912477 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.912439    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b635b45b-f504-4cea-be9f-26568921a553-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:03.912697 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.912534    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6x6q\" (UniqueName: \"kubernetes.io/projected/776cf4b4-81f0-4db9-8188-65d6c329f512-kube-api-access-v6x6q\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:03.912697 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.912560    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776cf4b4-81f0-4db9-8188-65d6c329f512-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:03.912697 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.912590    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:03.912697 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.912617    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:03.912933 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.912746    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:03.912933 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.912794    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:03.912933 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.912835    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wznfs\" (UniqueName: \"kubernetes.io/projected/b635b45b-f504-4cea-be9f-26568921a553-kube-api-access-wznfs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:03.912933 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.912900    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:03.913145 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.912937    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:03.913145 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.912965    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:03.913145 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.912988    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:03.913145 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.913046    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:03.913404 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.913190    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:03.913404 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.913394    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:03.913515 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.913394    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:03.913515 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.913462    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:03.913623 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.913581    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:03.915447 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.915424    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b635b45b-f504-4cea-be9f-26568921a553-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:03.915591 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.915549    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:03.915640 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.915619    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:03.915822 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.915805    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776cf4b4-81f0-4db9-8188-65d6c329f512-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:03.921900 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.921876    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6x6q\" (UniqueName: \"kubernetes.io/projected/776cf4b4-81f0-4db9-8188-65d6c329f512-kube-api-access-v6x6q\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:03.922068 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.922050    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wznfs\" (UniqueName: \"kubernetes.io/projected/b635b45b-f504-4cea-be9f-26568921a553-kube-api-access-wznfs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:03.969238 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.969145    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr"]
Jul 13 20:08:03.974552 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.974525    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr"
Jul 13 20:08:03.977334 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.977272    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-lznbq\""
Jul 13 20:08:03.979918 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:03.979889    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr"]
Jul 13 20:08:04.018257 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.018210    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:04.030484 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.030448    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:04.115396 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.115362    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr\" (UID: \"0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr"
Jul 13 20:08:04.115566 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.115415    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j27hg\" (UniqueName: \"kubernetes.io/projected/0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e-kube-api-access-j27hg\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr\" (UID: \"0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr"
Jul 13 20:08:04.115566 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.115521    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr\" (UID: \"0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr"
Jul 13 20:08:04.176198 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.176170    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"]
Jul 13 20:08:04.179000 ip-10-0-139-134 kubenswrapper[2569]: W0713 20:08:04.178969    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776cf4b4_81f0_4db9_8188_65d6c329f512.slice/crio-a4a18c69b2e7939f24ec2e715ea37ec962cc5448cb3eef2499e4ac199ed04b10 WatchSource:0}: Error finding container a4a18c69b2e7939f24ec2e715ea37ec962cc5448cb3eef2499e4ac199ed04b10: Status 404 returned error can't find the container with id a4a18c69b2e7939f24ec2e715ea37ec962cc5448cb3eef2499e4ac199ed04b10
Jul 13 20:08:04.204370 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.204340    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"]
Jul 13 20:08:04.216682 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.216652    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr\" (UID: \"0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr"
Jul 13 20:08:04.216860 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.216691    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j27hg\" (UniqueName: \"kubernetes.io/projected/0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e-kube-api-access-j27hg\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr\" (UID: \"0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr"
Jul 13 20:08:04.216860 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.216744    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr\" (UID: \"0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr"
Jul 13 20:08:04.217059 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.217035    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr\" (UID: \"0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr"
Jul 13 20:08:04.219621 ip-10-0-139-134 kubenswrapper[2569]: W0713 20:08:04.219547    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb635b45b_f504_4cea_be9f_26568921a553.slice/crio-753c1cc4cb1abc240e17b72bba40b2f588e3ce213cc301ee63ec33462f4c0663 WatchSource:0}: Error finding container 753c1cc4cb1abc240e17b72bba40b2f588e3ce213cc301ee63ec33462f4c0663: Status 404 returned error can't find the container with id 753c1cc4cb1abc240e17b72bba40b2f588e3ce213cc301ee63ec33462f4c0663
Jul 13 20:08:04.219725 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.219708    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr\" (UID: \"0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr"
Jul 13 20:08:04.226142 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.226118    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j27hg\" (UniqueName: \"kubernetes.io/projected/0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e-kube-api-access-j27hg\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr\" (UID: \"0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr"
Jul 13 20:08:04.287031 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.286999    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr"
Jul 13 20:08:04.429045 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.429017    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr"]
Jul 13 20:08:04.431691 ip-10-0-139-134 kubenswrapper[2569]: W0713 20:08:04.431661    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5bf7e3_8c1c_4cf1_9828_e740a6e8ca7e.slice/crio-0ac6d6af8f4e1f09ba17ade2c3c6ffda15c00745d4bca63a82cc1bfc48e8ffa6 WatchSource:0}: Error finding container 0ac6d6af8f4e1f09ba17ade2c3c6ffda15c00745d4bca63a82cc1bfc48e8ffa6: Status 404 returned error can't find the container with id 0ac6d6af8f4e1f09ba17ade2c3c6ffda15c00745d4bca63a82cc1bfc48e8ffa6
Jul 13 20:08:04.668260 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.668221    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9" event={"ID":"776cf4b4-81f0-4db9-8188-65d6c329f512","Type":"ContainerStarted","Data":"4ec3a13df17c5db1c64af9b26a9a7141f8049b07e1c2bf008bb133d3a4086474"}
Jul 13 20:08:04.668550 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.668319    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9" event={"ID":"776cf4b4-81f0-4db9-8188-65d6c329f512","Type":"ContainerStarted","Data":"a4a18c69b2e7939f24ec2e715ea37ec962cc5448cb3eef2499e4ac199ed04b10"}
Jul 13 20:08:04.668550 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.668404    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:04.669975 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.669951    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m" event={"ID":"b635b45b-f504-4cea-be9f-26568921a553","Type":"ContainerStarted","Data":"653a864b94125b320bf78ce5c7a9a6625fe0b084aef3dec7aec60337c9670ef0"}
Jul 13 20:08:04.670099 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.669993    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m" event={"ID":"b635b45b-f504-4cea-be9f-26568921a553","Type":"ContainerStarted","Data":"753c1cc4cb1abc240e17b72bba40b2f588e3ce213cc301ee63ec33462f4c0663"}
Jul 13 20:08:04.671438 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.671420    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr" event={"ID":"0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e","Type":"ContainerStarted","Data":"3d96c211a0241bc6a44bedb2c1e597b6b79f28e7fc8542e79a3911f756d83496"}
Jul 13 20:08:04.671520 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.671452    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr" event={"ID":"0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e","Type":"ContainerStarted","Data":"0ac6d6af8f4e1f09ba17ade2c3c6ffda15c00745d4bca63a82cc1bfc48e8ffa6"}
Jul 13 20:08:04.671520 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.671477    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr"
Jul 13 20:08:04.709318 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.709250    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m" podStartSLOduration=1.709234204 podStartE2EDuration="1.709234204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:08:04.708433336 +0000 UTC m=+4706.222232505" watchObservedRunningTime="2026-07-13 20:08:04.709234204 +0000 UTC m=+4706.223033369"
Jul 13 20:08:04.733657 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:04.733604    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr" podStartSLOduration=1.733588403 podStartE2EDuration="1.733588403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:08:04.732140266 +0000 UTC m=+4706.245939434" watchObservedRunningTime="2026-07-13 20:08:04.733588403 +0000 UTC m=+4706.247387583"
Jul 13 20:08:05.031030 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:05.030941    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:05.682577 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:05.682533    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9" event={"ID":"776cf4b4-81f0-4db9-8188-65d6c329f512","Type":"ContainerStarted","Data":"f375cf6084ab66305ddca4c5c9574fb90d0971fd38e70b4fd475062cdc659e42"}
Jul 13 20:08:05.703832 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:05.703777    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9" podStartSLOduration=2.703755745 podStartE2EDuration="2.703755745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:08:05.702410144 +0000 UTC m=+4707.216209312" watchObservedRunningTime="2026-07-13 20:08:05.703755745 +0000 UTC m=+4707.217554912"
Jul 13 20:08:06.018748 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:06.018657    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:14.018754 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:14.018708    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:14.031620 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:14.031583    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:14.031939 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:14.031918    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:14.037411 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:14.037390    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:14.044981 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:14.044959    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:14.738384 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:14.738360    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:08:14.739181 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:14.739163    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:08:35.684872 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:08:35.684843    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr"
Jul 13 20:09:32.059470 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:32.059432    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"]
Jul 13 20:09:32.061918 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:32.061872    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"]
Jul 13 20:09:32.063881 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:32.063852    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr"]
Jul 13 20:09:32.076653 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:09:32.076596    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 20:09:32.076653 ip-10-0-139-134 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:09:32.076653 ip-10-0-139-134 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 13 20:09:32.076653 ip-10-0-139-134 kubenswrapper[2569]:  > containerID="3d96c211a0241bc6a44bedb2c1e597b6b79f28e7fc8542e79a3911f756d83496" cmd=["/bin/sleep","15"]
Jul 13 20:09:32.076944 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:32.076671    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr" message="" err=<
Jul 13 20:09:32.076944 ip-10-0-139-134 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:09:32.076944 ip-10-0-139-134 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 13 20:09:32.076944 ip-10-0-139-134 kubenswrapper[2569]:  >
Jul 13 20:09:32.076944 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:09:32.076689    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 20:09:32.076944 ip-10-0-139-134 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:09:32.076944 ip-10-0-139-134 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 13 20:09:32.076944 ip-10-0-139-134 kubenswrapper[2569]:  > pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr" podUID="0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e" containerName="main" containerID="cri-o://3d96c211a0241bc6a44bedb2c1e597b6b79f28e7fc8542e79a3911f756d83496"
Jul 13 20:09:32.076944 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:32.076753    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr" podUID="0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e" containerName="main" containerID="cri-o://3d96c211a0241bc6a44bedb2c1e597b6b79f28e7fc8542e79a3911f756d83496" gracePeriod=60
Jul 13 20:09:32.347598 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:32.347568    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr"
Jul 13 20:09:32.421373 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:32.421329    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e-tls-certs\") pod \"0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e\" (UID: \"0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e\") "
Jul 13 20:09:32.421556 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:32.421399    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j27hg\" (UniqueName: \"kubernetes.io/projected/0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e-kube-api-access-j27hg\") pod \"0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e\" (UID: \"0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e\") "
Jul 13 20:09:32.421556 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:32.421478    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e-tokenizer-uds\") pod \"0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e\" (UID: \"0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e\") "
Jul 13 20:09:32.421769 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:32.421742    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e" (UID: "0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:09:32.423769 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:32.423731    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e" (UID: "0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:09:32.423881 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:32.423771    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e-kube-api-access-j27hg" (OuterVolumeSpecName: "kube-api-access-j27hg") pod "0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e" (UID: "0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e"). InnerVolumeSpecName "kube-api-access-j27hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:09:32.522465 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:32.522424    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j27hg\" (UniqueName: \"kubernetes.io/projected/0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e-kube-api-access-j27hg\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:09:32.522465 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:32.522459    2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e-tokenizer-uds\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:09:32.522465 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:32.522468    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e-tls-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:09:33.116449 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:33.116404    2569 generic.go:356] "Generic (PLEG): container finished" podID="0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e" containerID="3d96c211a0241bc6a44bedb2c1e597b6b79f28e7fc8542e79a3911f756d83496" exitCode=0
Jul 13 20:09:33.116894 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:33.116479    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr"
Jul 13 20:09:33.116894 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:33.116488    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr" event={"ID":"0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e","Type":"ContainerDied","Data":"3d96c211a0241bc6a44bedb2c1e597b6b79f28e7fc8542e79a3911f756d83496"}
Jul 13 20:09:33.116894 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:33.116570    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr" event={"ID":"0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e","Type":"ContainerDied","Data":"0ac6d6af8f4e1f09ba17ade2c3c6ffda15c00745d4bca63a82cc1bfc48e8ffa6"}
Jul 13 20:09:33.116894 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:33.116597    2569 scope.go:117] "RemoveContainer" containerID="3d96c211a0241bc6a44bedb2c1e597b6b79f28e7fc8542e79a3911f756d83496"
Jul 13 20:09:33.126545 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:33.126519    2569 scope.go:117] "RemoveContainer" containerID="3d96c211a0241bc6a44bedb2c1e597b6b79f28e7fc8542e79a3911f756d83496"
Jul 13 20:09:33.126937 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:09:33.126881    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d96c211a0241bc6a44bedb2c1e597b6b79f28e7fc8542e79a3911f756d83496\": container with ID starting with 3d96c211a0241bc6a44bedb2c1e597b6b79f28e7fc8542e79a3911f756d83496 not found: ID does not exist" containerID="3d96c211a0241bc6a44bedb2c1e597b6b79f28e7fc8542e79a3911f756d83496"
Jul 13 20:09:33.127059 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:33.126946    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d96c211a0241bc6a44bedb2c1e597b6b79f28e7fc8542e79a3911f756d83496"} err="failed to get container status \"3d96c211a0241bc6a44bedb2c1e597b6b79f28e7fc8542e79a3911f756d83496\": rpc error: code = NotFound desc = could not find container \"3d96c211a0241bc6a44bedb2c1e597b6b79f28e7fc8542e79a3911f756d83496\": container with ID starting with 3d96c211a0241bc6a44bedb2c1e597b6b79f28e7fc8542e79a3911f756d83496 not found: ID does not exist"
Jul 13 20:09:33.133586 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:33.133556    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr"]
Jul 13 20:09:33.137094 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:33.137066    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedz7ptr"]
Jul 13 20:09:35.060243 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:35.060209    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e" path="/var/lib/kubelet/pods/0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e/volumes"
Jul 13 20:09:39.355071 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:39.355039    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 20:09:39.360231 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:39.360205    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 20:09:39.367557 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:39.367527    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 20:09:39.372519 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:39.372493    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 20:09:47.086760 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.086688    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9" podUID="776cf4b4-81f0-4db9-8188-65d6c329f512" containerName="main" containerID="cri-o://f375cf6084ab66305ddca4c5c9574fb90d0971fd38e70b4fd475062cdc659e42" gracePeriod=45
Jul 13 20:09:47.087488 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.087324    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m" podUID="b635b45b-f504-4cea-be9f-26568921a553" containerName="main" containerID="cri-o://653a864b94125b320bf78ce5c7a9a6625fe0b084aef3dec7aec60337c9670ef0" gracePeriod=45
Jul 13 20:09:47.204662 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.204627    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9" podUID="776cf4b4-81f0-4db9-8188-65d6c329f512" containerName="llm-d-routing-sidecar" containerID="cri-o://4ec3a13df17c5db1c64af9b26a9a7141f8049b07e1c2bf008bb133d3a4086474" gracePeriod=45
Jul 13 20:09:47.370327 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.370279    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:09:47.454971 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.454929    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b635b45b-f504-4cea-be9f-26568921a553-tls-certs\") pod \"b635b45b-f504-4cea-be9f-26568921a553\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") "
Jul 13 20:09:47.455182 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.454996    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-tmp-dir\") pod \"b635b45b-f504-4cea-be9f-26568921a553\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") "
Jul 13 20:09:47.455182 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.455048    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-model-cache\") pod \"b635b45b-f504-4cea-be9f-26568921a553\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") "
Jul 13 20:09:47.455182 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.455071    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-dshm\") pod \"b635b45b-f504-4cea-be9f-26568921a553\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") "
Jul 13 20:09:47.455182 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.455094    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wznfs\" (UniqueName: \"kubernetes.io/projected/b635b45b-f504-4cea-be9f-26568921a553-kube-api-access-wznfs\") pod \"b635b45b-f504-4cea-be9f-26568921a553\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") "
Jul 13 20:09:47.455182 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.455147    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-home\") pod \"b635b45b-f504-4cea-be9f-26568921a553\" (UID: \"b635b45b-f504-4cea-be9f-26568921a553\") "
Jul 13 20:09:47.455496 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.455272    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b635b45b-f504-4cea-be9f-26568921a553" (UID: "b635b45b-f504-4cea-be9f-26568921a553"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:09:47.455496 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.455281    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-model-cache" (OuterVolumeSpecName: "model-cache") pod "b635b45b-f504-4cea-be9f-26568921a553" (UID: "b635b45b-f504-4cea-be9f-26568921a553"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:09:47.455496 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.455442    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-model-cache\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:09:47.455496 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.455456    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-tmp-dir\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:09:47.455708 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.455535    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-home" (OuterVolumeSpecName: "home") pod "b635b45b-f504-4cea-be9f-26568921a553" (UID: "b635b45b-f504-4cea-be9f-26568921a553"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:09:47.457425 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.457393    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b635b45b-f504-4cea-be9f-26568921a553-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b635b45b-f504-4cea-be9f-26568921a553" (UID: "b635b45b-f504-4cea-be9f-26568921a553"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:09:47.457551 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.457470    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b635b45b-f504-4cea-be9f-26568921a553-kube-api-access-wznfs" (OuterVolumeSpecName: "kube-api-access-wznfs") pod "b635b45b-f504-4cea-be9f-26568921a553" (UID: "b635b45b-f504-4cea-be9f-26568921a553"). InnerVolumeSpecName "kube-api-access-wznfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:09:47.457782 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.457761    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-dshm" (OuterVolumeSpecName: "dshm") pod "b635b45b-f504-4cea-be9f-26568921a553" (UID: "b635b45b-f504-4cea-be9f-26568921a553"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:09:47.471522 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.471494    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:09:47.555793 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.555752    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-home\") pod \"776cf4b4-81f0-4db9-8188-65d6c329f512\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") "
Jul 13 20:09:47.555793 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.555792    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-model-cache\") pod \"776cf4b4-81f0-4db9-8188-65d6c329f512\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") "
Jul 13 20:09:47.556055 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.555832    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776cf4b4-81f0-4db9-8188-65d6c329f512-tls-certs\") pod \"776cf4b4-81f0-4db9-8188-65d6c329f512\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") "
Jul 13 20:09:47.556055 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.555892    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-tmp-dir\") pod \"776cf4b4-81f0-4db9-8188-65d6c329f512\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") "
Jul 13 20:09:47.556055 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.555920    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6x6q\" (UniqueName: \"kubernetes.io/projected/776cf4b4-81f0-4db9-8188-65d6c329f512-kube-api-access-v6x6q\") pod \"776cf4b4-81f0-4db9-8188-65d6c329f512\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") "
Jul 13 20:09:47.556055 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.555988    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-dshm\") pod \"776cf4b4-81f0-4db9-8188-65d6c329f512\" (UID: \"776cf4b4-81f0-4db9-8188-65d6c329f512\") "
Jul 13 20:09:47.556275 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.556048    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-home" (OuterVolumeSpecName: "home") pod "776cf4b4-81f0-4db9-8188-65d6c329f512" (UID: "776cf4b4-81f0-4db9-8188-65d6c329f512"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:09:47.556275 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.556152    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-model-cache" (OuterVolumeSpecName: "model-cache") pod "776cf4b4-81f0-4db9-8188-65d6c329f512" (UID: "776cf4b4-81f0-4db9-8188-65d6c329f512"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:09:47.556275 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.556167    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "776cf4b4-81f0-4db9-8188-65d6c329f512" (UID: "776cf4b4-81f0-4db9-8188-65d6c329f512"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:09:47.556489 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.556304    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-dshm\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:09:47.556489 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.556325    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wznfs\" (UniqueName: \"kubernetes.io/projected/b635b45b-f504-4cea-be9f-26568921a553-kube-api-access-wznfs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:09:47.556489 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.556339    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b635b45b-f504-4cea-be9f-26568921a553-home\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:09:47.556489 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.556353    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-home\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:09:47.556489 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.556363    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-model-cache\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:09:47.556489 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.556371    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b635b45b-f504-4cea-be9f-26568921a553-tls-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:09:47.556489 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.556378    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-tmp-dir\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:09:47.558142 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.558120    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/776cf4b4-81f0-4db9-8188-65d6c329f512-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "776cf4b4-81f0-4db9-8188-65d6c329f512" (UID: "776cf4b4-81f0-4db9-8188-65d6c329f512"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:09:47.558747 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.558723    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-dshm" (OuterVolumeSpecName: "dshm") pod "776cf4b4-81f0-4db9-8188-65d6c329f512" (UID: "776cf4b4-81f0-4db9-8188-65d6c329f512"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:09:47.558747 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.558740    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776cf4b4-81f0-4db9-8188-65d6c329f512-kube-api-access-v6x6q" (OuterVolumeSpecName: "kube-api-access-v6x6q") pod "776cf4b4-81f0-4db9-8188-65d6c329f512" (UID: "776cf4b4-81f0-4db9-8188-65d6c329f512"). InnerVolumeSpecName "kube-api-access-v6x6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:09:47.657159 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.657063    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776cf4b4-81f0-4db9-8188-65d6c329f512-dshm\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:09:47.657159 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.657096    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776cf4b4-81f0-4db9-8188-65d6c329f512-tls-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:09:47.657159 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:47.657110    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6x6q\" (UniqueName: \"kubernetes.io/projected/776cf4b4-81f0-4db9-8188-65d6c329f512-kube-api-access-v6x6q\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:09:48.190032 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.190000    2569 generic.go:356] "Generic (PLEG): container finished" podID="b635b45b-f504-4cea-be9f-26568921a553" containerID="653a864b94125b320bf78ce5c7a9a6625fe0b084aef3dec7aec60337c9670ef0" exitCode=0
Jul 13 20:09:48.190487 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.190051    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"
Jul 13 20:09:48.190487 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.190091    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m" event={"ID":"b635b45b-f504-4cea-be9f-26568921a553","Type":"ContainerDied","Data":"653a864b94125b320bf78ce5c7a9a6625fe0b084aef3dec7aec60337c9670ef0"}
Jul 13 20:09:48.190487 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.190145    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m" event={"ID":"b635b45b-f504-4cea-be9f-26568921a553","Type":"ContainerDied","Data":"753c1cc4cb1abc240e17b72bba40b2f588e3ce213cc301ee63ec33462f4c0663"}
Jul 13 20:09:48.190487 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.190174    2569 scope.go:117] "RemoveContainer" containerID="653a864b94125b320bf78ce5c7a9a6625fe0b084aef3dec7aec60337c9670ef0"
Jul 13 20:09:48.192118 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.192090    2569 generic.go:356] "Generic (PLEG): container finished" podID="776cf4b4-81f0-4db9-8188-65d6c329f512" containerID="f375cf6084ab66305ddca4c5c9574fb90d0971fd38e70b4fd475062cdc659e42" exitCode=0
Jul 13 20:09:48.192118 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.192118    2569 generic.go:356] "Generic (PLEG): container finished" podID="776cf4b4-81f0-4db9-8188-65d6c329f512" containerID="4ec3a13df17c5db1c64af9b26a9a7141f8049b07e1c2bf008bb133d3a4086474" exitCode=0
Jul 13 20:09:48.192326 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.192141    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9" event={"ID":"776cf4b4-81f0-4db9-8188-65d6c329f512","Type":"ContainerDied","Data":"f375cf6084ab66305ddca4c5c9574fb90d0971fd38e70b4fd475062cdc659e42"}
Jul 13 20:09:48.192326 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.192150    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"
Jul 13 20:09:48.192326 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.192186    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9" event={"ID":"776cf4b4-81f0-4db9-8188-65d6c329f512","Type":"ContainerDied","Data":"4ec3a13df17c5db1c64af9b26a9a7141f8049b07e1c2bf008bb133d3a4086474"}
Jul 13 20:09:48.192326 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.192212    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9" event={"ID":"776cf4b4-81f0-4db9-8188-65d6c329f512","Type":"ContainerDied","Data":"a4a18c69b2e7939f24ec2e715ea37ec962cc5448cb3eef2499e4ac199ed04b10"}
Jul 13 20:09:48.199947 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.199929    2569 scope.go:117] "RemoveContainer" containerID="653a864b94125b320bf78ce5c7a9a6625fe0b084aef3dec7aec60337c9670ef0"
Jul 13 20:09:48.200252 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:09:48.200233    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"653a864b94125b320bf78ce5c7a9a6625fe0b084aef3dec7aec60337c9670ef0\": container with ID starting with 653a864b94125b320bf78ce5c7a9a6625fe0b084aef3dec7aec60337c9670ef0 not found: ID does not exist" containerID="653a864b94125b320bf78ce5c7a9a6625fe0b084aef3dec7aec60337c9670ef0"
Jul 13 20:09:48.200358 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.200263    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"653a864b94125b320bf78ce5c7a9a6625fe0b084aef3dec7aec60337c9670ef0"} err="failed to get container status \"653a864b94125b320bf78ce5c7a9a6625fe0b084aef3dec7aec60337c9670ef0\": rpc error: code = NotFound desc = could not find container \"653a864b94125b320bf78ce5c7a9a6625fe0b084aef3dec7aec60337c9670ef0\": container with ID starting with 653a864b94125b320bf78ce5c7a9a6625fe0b084aef3dec7aec60337c9670ef0 not found: ID does not exist"
Jul 13 20:09:48.200358 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.200280    2569 scope.go:117] "RemoveContainer" containerID="f375cf6084ab66305ddca4c5c9574fb90d0971fd38e70b4fd475062cdc659e42"
Jul 13 20:09:48.209369 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.209343    2569 scope.go:117] "RemoveContainer" containerID="4ec3a13df17c5db1c64af9b26a9a7141f8049b07e1c2bf008bb133d3a4086474"
Jul 13 20:09:48.216993 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.216963    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"]
Jul 13 20:09:48.218815 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.218697    2569 scope.go:117] "RemoveContainer" containerID="f375cf6084ab66305ddca4c5c9574fb90d0971fd38e70b4fd475062cdc659e42"
Jul 13 20:09:48.219107 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:09:48.219079    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f375cf6084ab66305ddca4c5c9574fb90d0971fd38e70b4fd475062cdc659e42\": container with ID starting with f375cf6084ab66305ddca4c5c9574fb90d0971fd38e70b4fd475062cdc659e42 not found: ID does not exist" containerID="f375cf6084ab66305ddca4c5c9574fb90d0971fd38e70b4fd475062cdc659e42"
Jul 13 20:09:48.219212 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.219119    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f375cf6084ab66305ddca4c5c9574fb90d0971fd38e70b4fd475062cdc659e42"} err="failed to get container status \"f375cf6084ab66305ddca4c5c9574fb90d0971fd38e70b4fd475062cdc659e42\": rpc error: code = NotFound desc = could not find container \"f375cf6084ab66305ddca4c5c9574fb90d0971fd38e70b4fd475062cdc659e42\": container with ID starting with f375cf6084ab66305ddca4c5c9574fb90d0971fd38e70b4fd475062cdc659e42 not found: ID does not exist"
Jul 13 20:09:48.219212 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.219144    2569 scope.go:117] "RemoveContainer" containerID="4ec3a13df17c5db1c64af9b26a9a7141f8049b07e1c2bf008bb133d3a4086474"
Jul 13 20:09:48.219551 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:09:48.219524    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec3a13df17c5db1c64af9b26a9a7141f8049b07e1c2bf008bb133d3a4086474\": container with ID starting with 4ec3a13df17c5db1c64af9b26a9a7141f8049b07e1c2bf008bb133d3a4086474 not found: ID does not exist" containerID="4ec3a13df17c5db1c64af9b26a9a7141f8049b07e1c2bf008bb133d3a4086474"
Jul 13 20:09:48.219661 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.219558    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec3a13df17c5db1c64af9b26a9a7141f8049b07e1c2bf008bb133d3a4086474"} err="failed to get container status \"4ec3a13df17c5db1c64af9b26a9a7141f8049b07e1c2bf008bb133d3a4086474\": rpc error: code = NotFound desc = could not find container \"4ec3a13df17c5db1c64af9b26a9a7141f8049b07e1c2bf008bb133d3a4086474\": container with ID starting with 4ec3a13df17c5db1c64af9b26a9a7141f8049b07e1c2bf008bb133d3a4086474 not found: ID does not exist"
Jul 13 20:09:48.219661 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.219581    2569 scope.go:117] "RemoveContainer" containerID="f375cf6084ab66305ddca4c5c9574fb90d0971fd38e70b4fd475062cdc659e42"
Jul 13 20:09:48.219858 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.219842    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f375cf6084ab66305ddca4c5c9574fb90d0971fd38e70b4fd475062cdc659e42"} err="failed to get container status \"f375cf6084ab66305ddca4c5c9574fb90d0971fd38e70b4fd475062cdc659e42\": rpc error: code = NotFound desc = could not find container \"f375cf6084ab66305ddca4c5c9574fb90d0971fd38e70b4fd475062cdc659e42\": container with ID starting with f375cf6084ab66305ddca4c5c9574fb90d0971fd38e70b4fd475062cdc659e42 not found: ID does not exist"
Jul 13 20:09:48.219916 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.219858    2569 scope.go:117] "RemoveContainer" containerID="4ec3a13df17c5db1c64af9b26a9a7141f8049b07e1c2bf008bb133d3a4086474"
Jul 13 20:09:48.220127 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.220101    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec3a13df17c5db1c64af9b26a9a7141f8049b07e1c2bf008bb133d3a4086474"} err="failed to get container status \"4ec3a13df17c5db1c64af9b26a9a7141f8049b07e1c2bf008bb133d3a4086474\": rpc error: code = NotFound desc = could not find container \"4ec3a13df17c5db1c64af9b26a9a7141f8049b07e1c2bf008bb133d3a4086474\": container with ID starting with 4ec3a13df17c5db1c64af9b26a9a7141f8049b07e1c2bf008bb133d3a4086474 not found: ID does not exist"
Jul 13 20:09:48.220512 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.220493    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5ncl6m"]
Jul 13 20:09:48.230679 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.230646    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"]
Jul 13 20:09:48.233931 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:48.233903    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-t8xj9"]
Jul 13 20:09:49.061164 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:49.061123    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="776cf4b4-81f0-4db9-8188-65d6c329f512" path="/var/lib/kubelet/pods/776cf4b4-81f0-4db9-8188-65d6c329f512/volumes"
Jul 13 20:09:49.061684 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:09:49.061665    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b635b45b-f504-4cea-be9f-26568921a553" path="/var/lib/kubelet/pods/b635b45b-f504-4cea-be9f-26568921a553/volumes"
Jul 13 20:14:39.386586 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:14:39.386481    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 20:14:39.390752 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:14:39.390731    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 20:14:39.397206 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:14:39.397186    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 20:14:39.400946 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:14:39.400931    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 20:15:42.774706 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.774672    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"]
Jul 13 20:15:42.775342 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.774995    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776cf4b4-81f0-4db9-8188-65d6c329f512" containerName="main"
Jul 13 20:15:42.775342 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.775005    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="776cf4b4-81f0-4db9-8188-65d6c329f512" containerName="main"
Jul 13 20:15:42.775342 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.775014    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776cf4b4-81f0-4db9-8188-65d6c329f512" containerName="llm-d-routing-sidecar"
Jul 13 20:15:42.775342 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.775022    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="776cf4b4-81f0-4db9-8188-65d6c329f512" containerName="llm-d-routing-sidecar"
Jul 13 20:15:42.775342 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.775037    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b635b45b-f504-4cea-be9f-26568921a553" containerName="main"
Jul 13 20:15:42.775342 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.775043    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b635b45b-f504-4cea-be9f-26568921a553" containerName="main"
Jul 13 20:15:42.775342 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.775048    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e" containerName="main"
Jul 13 20:15:42.775342 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.775052    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e" containerName="main"
Jul 13 20:15:42.775342 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.775111    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b635b45b-f504-4cea-be9f-26568921a553" containerName="main"
Jul 13 20:15:42.775342 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.775120    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="776cf4b4-81f0-4db9-8188-65d6c329f512" containerName="llm-d-routing-sidecar"
Jul 13 20:15:42.775342 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.775126    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="776cf4b4-81f0-4db9-8188-65d6c329f512" containerName="main"
Jul 13 20:15:42.775342 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.775134    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b5bf7e3-8c1c-4cf1-9828-e740a6e8ca7e" containerName="main"
Jul 13 20:15:42.778132 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.778115    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:42.781062 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.781030    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 13 20:15:42.788029 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.788005    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"]
Jul 13 20:15:42.923260 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.923219    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-xjzqz\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:42.923260 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.923264    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-xjzqz\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:42.923522 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.923323    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0acdf9d1-17c8-4f06-a31d-762e9b910124-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-xjzqz\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:42.923522 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.923418    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8554\" (UniqueName: \"kubernetes.io/projected/0acdf9d1-17c8-4f06-a31d-762e9b910124-kube-api-access-j8554\") pod \"rolling-upgrade-test-kserve-54bc76f55b-xjzqz\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:42.923522 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.923459    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-xjzqz\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:42.923522 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.923493    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-xjzqz\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:42.989768 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.989734    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm"]
Jul 13 20:15:42.993167 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.993133    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm"
Jul 13 20:15:42.995970 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:42.995946    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-epp-sa-dockercfg-hcf6g\""
Jul 13 20:15:43.006494 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.006460    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm"]
Jul 13 20:15:43.024548 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.024515    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8554\" (UniqueName: \"kubernetes.io/projected/0acdf9d1-17c8-4f06-a31d-762e9b910124-kube-api-access-j8554\") pod \"rolling-upgrade-test-kserve-54bc76f55b-xjzqz\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:43.024779 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.024571    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-xjzqz\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:43.024779 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.024616    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-xjzqz\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:43.024779 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.024664    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-xjzqz\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:43.024779 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.024702    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-xjzqz\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:43.024779 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.024735    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0acdf9d1-17c8-4f06-a31d-762e9b910124-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-xjzqz\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:43.025096 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.025044    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-xjzqz\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:43.025096 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.025060    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-xjzqz\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:43.025184 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.025159    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-xjzqz\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:43.027506 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.027480    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0acdf9d1-17c8-4f06-a31d-762e9b910124-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-xjzqz\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:43.027683 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.027581    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-xjzqz\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:43.038883 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.038853    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8554\" (UniqueName: \"kubernetes.io/projected/0acdf9d1-17c8-4f06-a31d-762e9b910124-kube-api-access-j8554\") pod \"rolling-upgrade-test-kserve-54bc76f55b-xjzqz\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:43.089002 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.088964    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:43.125430 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.125384    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/72b9de3d-df93-4f47-9586-d73a00ed171d-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm\" (UID: \"72b9de3d-df93-4f47-9586-d73a00ed171d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm"
Jul 13 20:15:43.125624 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.125551    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b9de3d-df93-4f47-9586-d73a00ed171d-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm\" (UID: \"72b9de3d-df93-4f47-9586-d73a00ed171d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm"
Jul 13 20:15:43.125624 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.125599    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x4rh\" (UniqueName: \"kubernetes.io/projected/72b9de3d-df93-4f47-9586-d73a00ed171d-kube-api-access-9x4rh\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm\" (UID: \"72b9de3d-df93-4f47-9586-d73a00ed171d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm"
Jul 13 20:15:43.226528 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.226494    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/72b9de3d-df93-4f47-9586-d73a00ed171d-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm\" (UID: \"72b9de3d-df93-4f47-9586-d73a00ed171d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm"
Jul 13 20:15:43.226725 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.226634    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b9de3d-df93-4f47-9586-d73a00ed171d-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm\" (UID: \"72b9de3d-df93-4f47-9586-d73a00ed171d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm"
Jul 13 20:15:43.226725 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.226661    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x4rh\" (UniqueName: \"kubernetes.io/projected/72b9de3d-df93-4f47-9586-d73a00ed171d-kube-api-access-9x4rh\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm\" (UID: \"72b9de3d-df93-4f47-9586-d73a00ed171d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm"
Jul 13 20:15:43.227386 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.227346    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/72b9de3d-df93-4f47-9586-d73a00ed171d-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm\" (UID: \"72b9de3d-df93-4f47-9586-d73a00ed171d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm"
Jul 13 20:15:43.227526 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.227409    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"]
Jul 13 20:15:43.229614 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.229591    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b9de3d-df93-4f47-9586-d73a00ed171d-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm\" (UID: \"72b9de3d-df93-4f47-9586-d73a00ed171d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm"
Jul 13 20:15:43.230887 ip-10-0-139-134 kubenswrapper[2569]: W0713 20:15:43.230865    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0acdf9d1_17c8_4f06_a31d_762e9b910124.slice/crio-b535ae6cdfe37d64b56adbb4f830a682a9fef437aa40272b709b3e6076e5fbcc WatchSource:0}: Error finding container b535ae6cdfe37d64b56adbb4f830a682a9fef437aa40272b709b3e6076e5fbcc: Status 404 returned error can't find the container with id b535ae6cdfe37d64b56adbb4f830a682a9fef437aa40272b709b3e6076e5fbcc
Jul 13 20:15:43.236016 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.235994    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x4rh\" (UniqueName: \"kubernetes.io/projected/72b9de3d-df93-4f47-9586-d73a00ed171d-kube-api-access-9x4rh\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm\" (UID: \"72b9de3d-df93-4f47-9586-d73a00ed171d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm"
Jul 13 20:15:43.305568 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.305535    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm"
Jul 13 20:15:43.444402 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.444373    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm"]
Jul 13 20:15:43.446715 ip-10-0-139-134 kubenswrapper[2569]: W0713 20:15:43.446687    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b9de3d_df93_4f47_9586_d73a00ed171d.slice/crio-e589b65aeb5f3b65f1163e11983de9e57e6e8fb6866cc1982a3f07b0aff389c9 WatchSource:0}: Error finding container e589b65aeb5f3b65f1163e11983de9e57e6e8fb6866cc1982a3f07b0aff389c9: Status 404 returned error can't find the container with id e589b65aeb5f3b65f1163e11983de9e57e6e8fb6866cc1982a3f07b0aff389c9
Jul 13 20:15:43.834417 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.834373    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz" event={"ID":"0acdf9d1-17c8-4f06-a31d-762e9b910124","Type":"ContainerStarted","Data":"bc26da1046c3478a430e1a01b6f59c5d9cd5dc4e8f45d7957182ca96065b2bf0"}
Jul 13 20:15:43.834889 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.834440    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz" event={"ID":"0acdf9d1-17c8-4f06-a31d-762e9b910124","Type":"ContainerStarted","Data":"b535ae6cdfe37d64b56adbb4f830a682a9fef437aa40272b709b3e6076e5fbcc"}
Jul 13 20:15:43.835958 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.835936    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm" event={"ID":"72b9de3d-df93-4f47-9586-d73a00ed171d","Type":"ContainerStarted","Data":"ea0911a04f7560ef01ad5e3df3752b0ce1694c346c2634744c93317af9bdce34"}
Jul 13 20:15:43.836080 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.835970    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm" event={"ID":"72b9de3d-df93-4f47-9586-d73a00ed171d","Type":"ContainerStarted","Data":"e589b65aeb5f3b65f1163e11983de9e57e6e8fb6866cc1982a3f07b0aff389c9"}
Jul 13 20:15:43.836080 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.836060    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm"
Jul 13 20:15:43.853631 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.853585    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz" podStartSLOduration=1.8535714859999999 podStartE2EDuration="1.853571486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:15:43.853157042 +0000 UTC m=+5165.366956208" watchObservedRunningTime="2026-07-13 20:15:43.853571486 +0000 UTC m=+5165.367370652"
Jul 13 20:15:43.871692 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:43.871642    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm" podStartSLOduration=1.871626444 podStartE2EDuration="1.871626444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:15:43.869929971 +0000 UTC m=+5165.383729138" watchObservedRunningTime="2026-07-13 20:15:43.871626444 +0000 UTC m=+5165.385425610"
Jul 13 20:15:44.089810 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:44.089785    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:53.089998 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:53.089960    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:53.102627 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:53.102601    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:15:53.897142 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:15:53.897109    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:16:14.841841 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:16:14.841806    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm"
Jul 13 20:17:18.873789 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:18.873751    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"]
Jul 13 20:17:18.877188 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:18.877169    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:18.888505 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:18.888479    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"]
Jul 13 20:17:18.984081 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:18.984040    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-7z6cd\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:18.984261 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:18.984112    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-7z6cd\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:18.984261 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:18.984173    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beae9f61-bc79-46a0-80dd-8a6b235c9c78-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-7z6cd\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:18.984261 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:18.984248    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-7z6cd\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:18.984431 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:18.984268    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bksnn\" (UniqueName: \"kubernetes.io/projected/beae9f61-bc79-46a0-80dd-8a6b235c9c78-kube-api-access-bksnn\") pod \"rolling-upgrade-test-kserve-859df57bcb-7z6cd\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:18.984431 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:18.984321    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-7z6cd\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:19.084777 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:19.084738    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-7z6cd\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:19.084777 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:19.084781    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beae9f61-bc79-46a0-80dd-8a6b235c9c78-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-7z6cd\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:19.085049 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:19.084830    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-7z6cd\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:19.085049 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:19.084853    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bksnn\" (UniqueName: \"kubernetes.io/projected/beae9f61-bc79-46a0-80dd-8a6b235c9c78-kube-api-access-bksnn\") pod \"rolling-upgrade-test-kserve-859df57bcb-7z6cd\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:19.085049 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:19.084877    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-7z6cd\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:19.085049 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:19.085029    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-7z6cd\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:19.085234 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:19.085210    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-7z6cd\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:19.085310 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:19.085273    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-7z6cd\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:19.085385 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:19.085371    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-7z6cd\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:19.087339 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:19.087316    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-7z6cd\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:19.087607 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:19.087588    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beae9f61-bc79-46a0-80dd-8a6b235c9c78-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-7z6cd\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:19.095987 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:19.095960    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bksnn\" (UniqueName: \"kubernetes.io/projected/beae9f61-bc79-46a0-80dd-8a6b235c9c78-kube-api-access-bksnn\") pod \"rolling-upgrade-test-kserve-859df57bcb-7z6cd\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:19.188320 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:19.188218    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:19.328828 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:19.328784    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"]
Jul 13 20:17:19.330198 ip-10-0-139-134 kubenswrapper[2569]: W0713 20:17:19.330163    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeae9f61_bc79_46a0_80dd_8a6b235c9c78.slice/crio-61ca98865581b71b0ef915815a174719e698cb13ccba1dc4b953b71273f3a5b7 WatchSource:0}: Error finding container 61ca98865581b71b0ef915815a174719e698cb13ccba1dc4b953b71273f3a5b7: Status 404 returned error can't find the container with id 61ca98865581b71b0ef915815a174719e698cb13ccba1dc4b953b71273f3a5b7
Jul 13 20:17:20.293612 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:20.293579    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd" event={"ID":"beae9f61-bc79-46a0-80dd-8a6b235c9c78","Type":"ContainerStarted","Data":"3dda40b7d54b9e1b07e9b0578ccdc3f28bd381c163f492e6c145e423413a7729"}
Jul 13 20:17:20.293999 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:20.293632    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd" event={"ID":"beae9f61-bc79-46a0-80dd-8a6b235c9c78","Type":"ContainerStarted","Data":"61ca98865581b71b0ef915815a174719e698cb13ccba1dc4b953b71273f3a5b7"}
Jul 13 20:17:20.313977 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:20.313925    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd" podStartSLOduration=2.31390914 podStartE2EDuration="2.31390914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:17:20.313373169 +0000 UTC m=+5261.827172336" watchObservedRunningTime="2026-07-13 20:17:20.31390914 +0000 UTC m=+5261.827708306"
Jul 13 20:17:21.188385 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:21.188345    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:29.188993 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:29.188955    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:29.201285 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:29.201252    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:29.352613 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:29.352579    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:29.463682 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:29.463591    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"]
Jul 13 20:17:38.742406 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:38.742368    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm"]
Jul 13 20:17:38.751752 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:38.751714    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"]
Jul 13 20:17:38.752950 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:17:38.752904    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 20:17:38.752950 ip-10-0-139-134 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:17:38.752950 ip-10-0-139-134 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 13 20:17:38.752950 ip-10-0-139-134 kubenswrapper[2569]:  > containerID="ea0911a04f7560ef01ad5e3df3752b0ce1694c346c2634744c93317af9bdce34" cmd=["/bin/sleep","15"]
Jul 13 20:17:38.753225 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:38.752966    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm" message="" err=<
Jul 13 20:17:38.753225 ip-10-0-139-134 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:17:38.753225 ip-10-0-139-134 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 13 20:17:38.753225 ip-10-0-139-134 kubenswrapper[2569]:  >
Jul 13 20:17:38.753225 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:17:38.752978    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 20:17:38.753225 ip-10-0-139-134 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 20:17:38.753225 ip-10-0-139-134 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 13 20:17:38.753225 ip-10-0-139-134 kubenswrapper[2569]:  > pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm" podUID="72b9de3d-df93-4f47-9586-d73a00ed171d" containerName="main" containerID="cri-o://ea0911a04f7560ef01ad5e3df3752b0ce1694c346c2634744c93317af9bdce34"
Jul 13 20:17:38.753225 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:38.753022    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm" podUID="72b9de3d-df93-4f47-9586-d73a00ed171d" containerName="main" containerID="cri-o://ea0911a04f7560ef01ad5e3df3752b0ce1694c346c2634744c93317af9bdce34" gracePeriod=60
Jul 13 20:17:39.014945 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:39.014912    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm"
Jul 13 20:17:39.058704 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:39.058670    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b9de3d-df93-4f47-9586-d73a00ed171d-tls-certs\") pod \"72b9de3d-df93-4f47-9586-d73a00ed171d\" (UID: \"72b9de3d-df93-4f47-9586-d73a00ed171d\") "
Jul 13 20:17:39.058902 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:39.058764    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/72b9de3d-df93-4f47-9586-d73a00ed171d-tokenizer-uds\") pod \"72b9de3d-df93-4f47-9586-d73a00ed171d\" (UID: \"72b9de3d-df93-4f47-9586-d73a00ed171d\") "
Jul 13 20:17:39.058902 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:39.058824    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x4rh\" (UniqueName: \"kubernetes.io/projected/72b9de3d-df93-4f47-9586-d73a00ed171d-kube-api-access-9x4rh\") pod \"72b9de3d-df93-4f47-9586-d73a00ed171d\" (UID: \"72b9de3d-df93-4f47-9586-d73a00ed171d\") "
Jul 13 20:17:39.059053 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:39.059010    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b9de3d-df93-4f47-9586-d73a00ed171d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "72b9de3d-df93-4f47-9586-d73a00ed171d" (UID: "72b9de3d-df93-4f47-9586-d73a00ed171d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:17:39.061620 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:39.061585    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b9de3d-df93-4f47-9586-d73a00ed171d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "72b9de3d-df93-4f47-9586-d73a00ed171d" (UID: "72b9de3d-df93-4f47-9586-d73a00ed171d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:17:39.061620 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:39.061593    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b9de3d-df93-4f47-9586-d73a00ed171d-kube-api-access-9x4rh" (OuterVolumeSpecName: "kube-api-access-9x4rh") pod "72b9de3d-df93-4f47-9586-d73a00ed171d" (UID: "72b9de3d-df93-4f47-9586-d73a00ed171d"). InnerVolumeSpecName "kube-api-access-9x4rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:17:39.159741 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:39.159703    2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/72b9de3d-df93-4f47-9586-d73a00ed171d-tokenizer-uds\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:17:39.159741 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:39.159736    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x4rh\" (UniqueName: \"kubernetes.io/projected/72b9de3d-df93-4f47-9586-d73a00ed171d-kube-api-access-9x4rh\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:17:39.159741 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:39.159746    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b9de3d-df93-4f47-9586-d73a00ed171d-tls-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:17:39.390525 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:39.390490    2569 generic.go:356] "Generic (PLEG): container finished" podID="72b9de3d-df93-4f47-9586-d73a00ed171d" containerID="ea0911a04f7560ef01ad5e3df3752b0ce1694c346c2634744c93317af9bdce34" exitCode=0
Jul 13 20:17:39.390704 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:39.390548    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm"
Jul 13 20:17:39.390704 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:39.390551    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm" event={"ID":"72b9de3d-df93-4f47-9586-d73a00ed171d","Type":"ContainerDied","Data":"ea0911a04f7560ef01ad5e3df3752b0ce1694c346c2634744c93317af9bdce34"}
Jul 13 20:17:39.390704 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:39.390671    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm" event={"ID":"72b9de3d-df93-4f47-9586-d73a00ed171d","Type":"ContainerDied","Data":"e589b65aeb5f3b65f1163e11983de9e57e6e8fb6866cc1982a3f07b0aff389c9"}
Jul 13 20:17:39.390704 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:39.390691    2569 scope.go:117] "RemoveContainer" containerID="ea0911a04f7560ef01ad5e3df3752b0ce1694c346c2634744c93317af9bdce34"
Jul 13 20:17:39.400215 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:39.400193    2569 scope.go:117] "RemoveContainer" containerID="ea0911a04f7560ef01ad5e3df3752b0ce1694c346c2634744c93317af9bdce34"
Jul 13 20:17:39.400551 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:17:39.400525    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea0911a04f7560ef01ad5e3df3752b0ce1694c346c2634744c93317af9bdce34\": container with ID starting with ea0911a04f7560ef01ad5e3df3752b0ce1694c346c2634744c93317af9bdce34 not found: ID does not exist" containerID="ea0911a04f7560ef01ad5e3df3752b0ce1694c346c2634744c93317af9bdce34"
Jul 13 20:17:39.400607 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:39.400560    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea0911a04f7560ef01ad5e3df3752b0ce1694c346c2634744c93317af9bdce34"} err="failed to get container status \"ea0911a04f7560ef01ad5e3df3752b0ce1694c346c2634744c93317af9bdce34\": rpc error: code = NotFound desc = could not find container \"ea0911a04f7560ef01ad5e3df3752b0ce1694c346c2634744c93317af9bdce34\": container with ID starting with ea0911a04f7560ef01ad5e3df3752b0ce1694c346c2634744c93317af9bdce34 not found: ID does not exist"
Jul 13 20:17:39.406834 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:39.406805    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm"]
Jul 13 20:17:39.413997 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:39.413967    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-dfnlm"]
Jul 13 20:17:41.061136 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:41.061096    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72b9de3d-df93-4f47-9586-d73a00ed171d" path="/var/lib/kubelet/pods/72b9de3d-df93-4f47-9586-d73a00ed171d/volumes"
Jul 13 20:17:44.485639 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:44.485594    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz" podUID="0acdf9d1-17c8-4f06-a31d-762e9b910124" containerName="main" containerID="cri-o://bc26da1046c3478a430e1a01b6f59c5d9cd5dc4e8f45d7957182ca96065b2bf0" gracePeriod=45
Jul 13 20:17:44.750085 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:44.750013    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:17:44.810864 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:44.810823    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-tmp-dir\") pod \"0acdf9d1-17c8-4f06-a31d-762e9b910124\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") "
Jul 13 20:17:44.810864 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:44.810863    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-dshm\") pod \"0acdf9d1-17c8-4f06-a31d-762e9b910124\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") "
Jul 13 20:17:44.811107 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:44.810901    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0acdf9d1-17c8-4f06-a31d-762e9b910124-tls-certs\") pod \"0acdf9d1-17c8-4f06-a31d-762e9b910124\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") "
Jul 13 20:17:44.811107 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:44.810932    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-home\") pod \"0acdf9d1-17c8-4f06-a31d-762e9b910124\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") "
Jul 13 20:17:44.811107 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:44.810956    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-model-cache\") pod \"0acdf9d1-17c8-4f06-a31d-762e9b910124\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") "
Jul 13 20:17:44.811107 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:44.810981    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8554\" (UniqueName: \"kubernetes.io/projected/0acdf9d1-17c8-4f06-a31d-762e9b910124-kube-api-access-j8554\") pod \"0acdf9d1-17c8-4f06-a31d-762e9b910124\" (UID: \"0acdf9d1-17c8-4f06-a31d-762e9b910124\") "
Jul 13 20:17:44.811342 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:44.811167    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0acdf9d1-17c8-4f06-a31d-762e9b910124" (UID: "0acdf9d1-17c8-4f06-a31d-762e9b910124"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:17:44.811342 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:44.811220    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-model-cache" (OuterVolumeSpecName: "model-cache") pod "0acdf9d1-17c8-4f06-a31d-762e9b910124" (UID: "0acdf9d1-17c8-4f06-a31d-762e9b910124"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:17:44.811342 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:44.811233    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-home" (OuterVolumeSpecName: "home") pod "0acdf9d1-17c8-4f06-a31d-762e9b910124" (UID: "0acdf9d1-17c8-4f06-a31d-762e9b910124"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:17:44.811342 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:44.811259    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-tmp-dir\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:17:44.813256 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:44.813230    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0acdf9d1-17c8-4f06-a31d-762e9b910124-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0acdf9d1-17c8-4f06-a31d-762e9b910124" (UID: "0acdf9d1-17c8-4f06-a31d-762e9b910124"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:17:44.813256 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:44.813245    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0acdf9d1-17c8-4f06-a31d-762e9b910124-kube-api-access-j8554" (OuterVolumeSpecName: "kube-api-access-j8554") pod "0acdf9d1-17c8-4f06-a31d-762e9b910124" (UID: "0acdf9d1-17c8-4f06-a31d-762e9b910124"). InnerVolumeSpecName "kube-api-access-j8554". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:17:44.813388 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:44.813240    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-dshm" (OuterVolumeSpecName: "dshm") pod "0acdf9d1-17c8-4f06-a31d-762e9b910124" (UID: "0acdf9d1-17c8-4f06-a31d-762e9b910124"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:17:44.912688 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:44.912648    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0acdf9d1-17c8-4f06-a31d-762e9b910124-tls-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:17:44.912688 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:44.912680    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-home\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:17:44.912688 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:44.912689    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-model-cache\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:17:44.912688 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:44.912698    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8554\" (UniqueName: \"kubernetes.io/projected/0acdf9d1-17c8-4f06-a31d-762e9b910124-kube-api-access-j8554\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:17:44.912994 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:44.912708    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0acdf9d1-17c8-4f06-a31d-762e9b910124-dshm\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:17:45.083603 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:17:45.083560    2569 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0acdf9d1_17c8_4f06_a31d_762e9b910124.slice/crio-b535ae6cdfe37d64b56adbb4f830a682a9fef437aa40272b709b3e6076e5fbcc\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0acdf9d1_17c8_4f06_a31d_762e9b910124.slice\": RecentStats: unable to find data in memory cache]"
Jul 13 20:17:45.417523 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:45.417483    2569 generic.go:356] "Generic (PLEG): container finished" podID="0acdf9d1-17c8-4f06-a31d-762e9b910124" containerID="bc26da1046c3478a430e1a01b6f59c5d9cd5dc4e8f45d7957182ca96065b2bf0" exitCode=0
Jul 13 20:17:45.417708 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:45.417532    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"
Jul 13 20:17:45.417708 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:45.417568    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz" event={"ID":"0acdf9d1-17c8-4f06-a31d-762e9b910124","Type":"ContainerDied","Data":"bc26da1046c3478a430e1a01b6f59c5d9cd5dc4e8f45d7957182ca96065b2bf0"}
Jul 13 20:17:45.417708 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:45.417632    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz" event={"ID":"0acdf9d1-17c8-4f06-a31d-762e9b910124","Type":"ContainerDied","Data":"b535ae6cdfe37d64b56adbb4f830a682a9fef437aa40272b709b3e6076e5fbcc"}
Jul 13 20:17:45.417708 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:45.417656    2569 scope.go:117] "RemoveContainer" containerID="bc26da1046c3478a430e1a01b6f59c5d9cd5dc4e8f45d7957182ca96065b2bf0"
Jul 13 20:17:45.426503 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:45.426481    2569 scope.go:117] "RemoveContainer" containerID="bc26da1046c3478a430e1a01b6f59c5d9cd5dc4e8f45d7957182ca96065b2bf0"
Jul 13 20:17:45.426833 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:17:45.426810    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc26da1046c3478a430e1a01b6f59c5d9cd5dc4e8f45d7957182ca96065b2bf0\": container with ID starting with bc26da1046c3478a430e1a01b6f59c5d9cd5dc4e8f45d7957182ca96065b2bf0 not found: ID does not exist" containerID="bc26da1046c3478a430e1a01b6f59c5d9cd5dc4e8f45d7957182ca96065b2bf0"
Jul 13 20:17:45.426934 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:45.426841    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc26da1046c3478a430e1a01b6f59c5d9cd5dc4e8f45d7957182ca96065b2bf0"} err="failed to get container status \"bc26da1046c3478a430e1a01b6f59c5d9cd5dc4e8f45d7957182ca96065b2bf0\": rpc error: code = NotFound desc = could not find container \"bc26da1046c3478a430e1a01b6f59c5d9cd5dc4e8f45d7957182ca96065b2bf0\": container with ID starting with bc26da1046c3478a430e1a01b6f59c5d9cd5dc4e8f45d7957182ca96065b2bf0 not found: ID does not exist"
Jul 13 20:17:45.435278 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:45.435243    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"]
Jul 13 20:17:45.439063 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:45.439030    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-xjzqz"]
Jul 13 20:17:47.060851 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:47.060802    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0acdf9d1-17c8-4f06-a31d-762e9b910124" path="/var/lib/kubelet/pods/0acdf9d1-17c8-4f06-a31d-762e9b910124/volumes"
Jul 13 20:17:53.774432 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:53.774388    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd" podUID="beae9f61-bc79-46a0-80dd-8a6b235c9c78" containerName="main" containerID="cri-o://3dda40b7d54b9e1b07e9b0578ccdc3f28bd381c163f492e6c145e423413a7729" gracePeriod=45
Jul 13 20:17:54.039876 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.039790    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:54.098197 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.098145    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bksnn\" (UniqueName: \"kubernetes.io/projected/beae9f61-bc79-46a0-80dd-8a6b235c9c78-kube-api-access-bksnn\") pod \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") "
Jul 13 20:17:54.098423 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.098262    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beae9f61-bc79-46a0-80dd-8a6b235c9c78-tls-certs\") pod \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") "
Jul 13 20:17:54.098423 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.098317    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-model-cache\") pod \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") "
Jul 13 20:17:54.098423 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.098363    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-tmp-dir\") pod \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") "
Jul 13 20:17:54.098423 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.098392    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-dshm\") pod \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") "
Jul 13 20:17:54.098423 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.098412    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-home\") pod \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\" (UID: \"beae9f61-bc79-46a0-80dd-8a6b235c9c78\") "
Jul 13 20:17:54.098653 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.098608    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-model-cache" (OuterVolumeSpecName: "model-cache") pod "beae9f61-bc79-46a0-80dd-8a6b235c9c78" (UID: "beae9f61-bc79-46a0-80dd-8a6b235c9c78"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:17:54.098877 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.098850    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-home" (OuterVolumeSpecName: "home") pod "beae9f61-bc79-46a0-80dd-8a6b235c9c78" (UID: "beae9f61-bc79-46a0-80dd-8a6b235c9c78"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:17:54.099019 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.098860    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "beae9f61-bc79-46a0-80dd-8a6b235c9c78" (UID: "beae9f61-bc79-46a0-80dd-8a6b235c9c78"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:17:54.100678 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.100647    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beae9f61-bc79-46a0-80dd-8a6b235c9c78-kube-api-access-bksnn" (OuterVolumeSpecName: "kube-api-access-bksnn") pod "beae9f61-bc79-46a0-80dd-8a6b235c9c78" (UID: "beae9f61-bc79-46a0-80dd-8a6b235c9c78"). InnerVolumeSpecName "kube-api-access-bksnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:17:54.101085 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.101058    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beae9f61-bc79-46a0-80dd-8a6b235c9c78-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "beae9f61-bc79-46a0-80dd-8a6b235c9c78" (UID: "beae9f61-bc79-46a0-80dd-8a6b235c9c78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:17:54.101085 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.101056    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-dshm" (OuterVolumeSpecName: "dshm") pod "beae9f61-bc79-46a0-80dd-8a6b235c9c78" (UID: "beae9f61-bc79-46a0-80dd-8a6b235c9c78"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:17:54.199676 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.199631    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bksnn\" (UniqueName: \"kubernetes.io/projected/beae9f61-bc79-46a0-80dd-8a6b235c9c78-kube-api-access-bksnn\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:17:54.199676 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.199676    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beae9f61-bc79-46a0-80dd-8a6b235c9c78-tls-certs\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:17:54.199930 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.199693    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-model-cache\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:17:54.199930 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.199707    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-tmp-dir\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:17:54.199930 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.199721    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-dshm\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:17:54.199930 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.199734    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beae9f61-bc79-46a0-80dd-8a6b235c9c78-home\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:17:54.468623 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.468590    2569 generic.go:356] "Generic (PLEG): container finished" podID="beae9f61-bc79-46a0-80dd-8a6b235c9c78" containerID="3dda40b7d54b9e1b07e9b0578ccdc3f28bd381c163f492e6c145e423413a7729" exitCode=0
Jul 13 20:17:54.468815 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.468647    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"
Jul 13 20:17:54.468815 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.468672    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd" event={"ID":"beae9f61-bc79-46a0-80dd-8a6b235c9c78","Type":"ContainerDied","Data":"3dda40b7d54b9e1b07e9b0578ccdc3f28bd381c163f492e6c145e423413a7729"}
Jul 13 20:17:54.468815 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.468757    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd" event={"ID":"beae9f61-bc79-46a0-80dd-8a6b235c9c78","Type":"ContainerDied","Data":"61ca98865581b71b0ef915815a174719e698cb13ccba1dc4b953b71273f3a5b7"}
Jul 13 20:17:54.468815 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.468782    2569 scope.go:117] "RemoveContainer" containerID="3dda40b7d54b9e1b07e9b0578ccdc3f28bd381c163f492e6c145e423413a7729"
Jul 13 20:17:54.478212 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.478188    2569 scope.go:117] "RemoveContainer" containerID="3dda40b7d54b9e1b07e9b0578ccdc3f28bd381c163f492e6c145e423413a7729"
Jul 13 20:17:54.478578 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:17:54.478558    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dda40b7d54b9e1b07e9b0578ccdc3f28bd381c163f492e6c145e423413a7729\": container with ID starting with 3dda40b7d54b9e1b07e9b0578ccdc3f28bd381c163f492e6c145e423413a7729 not found: ID does not exist" containerID="3dda40b7d54b9e1b07e9b0578ccdc3f28bd381c163f492e6c145e423413a7729"
Jul 13 20:17:54.478663 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.478591    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dda40b7d54b9e1b07e9b0578ccdc3f28bd381c163f492e6c145e423413a7729"} err="failed to get container status \"3dda40b7d54b9e1b07e9b0578ccdc3f28bd381c163f492e6c145e423413a7729\": rpc error: code = NotFound desc = could not find container \"3dda40b7d54b9e1b07e9b0578ccdc3f28bd381c163f492e6c145e423413a7729\": container with ID starting with 3dda40b7d54b9e1b07e9b0578ccdc3f28bd381c163f492e6c145e423413a7729 not found: ID does not exist"
Jul 13 20:17:54.491918 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.491879    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"]
Jul 13 20:17:54.497529 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:54.497501    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7z6cd"]
Jul 13 20:17:55.060722 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:55.060685    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="beae9f61-bc79-46a0-80dd-8a6b235c9c78" path="/var/lib/kubelet/pods/beae9f61-bc79-46a0-80dd-8a6b235c9c78/volumes"
Jul 13 20:17:58.494217 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:58.494179    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-dc6dd9667-pq4h9"]
Jul 13 20:17:58.494642 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:58.494525    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beae9f61-bc79-46a0-80dd-8a6b235c9c78" containerName="main"
Jul 13 20:17:58.494642 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:58.494537    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="beae9f61-bc79-46a0-80dd-8a6b235c9c78" containerName="main"
Jul 13 20:17:58.494642 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:58.494561    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0acdf9d1-17c8-4f06-a31d-762e9b910124" containerName="main"
Jul 13 20:17:58.494642 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:58.494568    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0acdf9d1-17c8-4f06-a31d-762e9b910124" containerName="main"
Jul 13 20:17:58.494642 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:58.494574    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72b9de3d-df93-4f47-9586-d73a00ed171d" containerName="main"
Jul 13 20:17:58.494642 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:58.494579    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="72b9de3d-df93-4f47-9586-d73a00ed171d" containerName="main"
Jul 13 20:17:58.494642 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:58.494634    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0acdf9d1-17c8-4f06-a31d-762e9b910124" containerName="main"
Jul 13 20:17:58.494642 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:58.494642    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="72b9de3d-df93-4f47-9586-d73a00ed171d" containerName="main"
Jul 13 20:17:58.494889 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:58.494650    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="beae9f61-bc79-46a0-80dd-8a6b235c9c78" containerName="main"
Jul 13 20:17:58.499063 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:58.499043    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-dc6dd9667-pq4h9"
Jul 13 20:17:58.507596 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:58.507565    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-dc6dd9667-pq4h9"]
Jul 13 20:17:58.643199 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:58.643156    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b63f5d1-ddc1-4b49-9bd1-69133ae2919f-cert\") pod \"llmisvc-controller-manager-dc6dd9667-pq4h9\" (UID: \"8b63f5d1-ddc1-4b49-9bd1-69133ae2919f\") " pod="kserve/llmisvc-controller-manager-dc6dd9667-pq4h9"
Jul 13 20:17:58.643424 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:58.643220    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq5km\" (UniqueName: \"kubernetes.io/projected/8b63f5d1-ddc1-4b49-9bd1-69133ae2919f-kube-api-access-fq5km\") pod \"llmisvc-controller-manager-dc6dd9667-pq4h9\" (UID: \"8b63f5d1-ddc1-4b49-9bd1-69133ae2919f\") " pod="kserve/llmisvc-controller-manager-dc6dd9667-pq4h9"
Jul 13 20:17:58.744114 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:58.744078    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq5km\" (UniqueName: \"kubernetes.io/projected/8b63f5d1-ddc1-4b49-9bd1-69133ae2919f-kube-api-access-fq5km\") pod \"llmisvc-controller-manager-dc6dd9667-pq4h9\" (UID: \"8b63f5d1-ddc1-4b49-9bd1-69133ae2919f\") " pod="kserve/llmisvc-controller-manager-dc6dd9667-pq4h9"
Jul 13 20:17:58.744365 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:58.744155    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b63f5d1-ddc1-4b49-9bd1-69133ae2919f-cert\") pod \"llmisvc-controller-manager-dc6dd9667-pq4h9\" (UID: \"8b63f5d1-ddc1-4b49-9bd1-69133ae2919f\") " pod="kserve/llmisvc-controller-manager-dc6dd9667-pq4h9"
Jul 13 20:17:58.746873 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:58.746840    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b63f5d1-ddc1-4b49-9bd1-69133ae2919f-cert\") pod \"llmisvc-controller-manager-dc6dd9667-pq4h9\" (UID: \"8b63f5d1-ddc1-4b49-9bd1-69133ae2919f\") " pod="kserve/llmisvc-controller-manager-dc6dd9667-pq4h9"
Jul 13 20:17:58.753214 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:58.753180    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq5km\" (UniqueName: \"kubernetes.io/projected/8b63f5d1-ddc1-4b49-9bd1-69133ae2919f-kube-api-access-fq5km\") pod \"llmisvc-controller-manager-dc6dd9667-pq4h9\" (UID: \"8b63f5d1-ddc1-4b49-9bd1-69133ae2919f\") " pod="kserve/llmisvc-controller-manager-dc6dd9667-pq4h9"
Jul 13 20:17:58.809377 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:58.809336    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-dc6dd9667-pq4h9"
Jul 13 20:17:58.953794 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:58.953767    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-dc6dd9667-pq4h9"]
Jul 13 20:17:58.956007 ip-10-0-139-134 kubenswrapper[2569]: W0713 20:17:58.955979    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8b63f5d1_ddc1_4b49_9bd1_69133ae2919f.slice/crio-1d90c95ff7eaf17cd5398813b23c446087c11a609e3d01ac3b395569235e1a7d WatchSource:0}: Error finding container 1d90c95ff7eaf17cd5398813b23c446087c11a609e3d01ac3b395569235e1a7d: Status 404 returned error can't find the container with id 1d90c95ff7eaf17cd5398813b23c446087c11a609e3d01ac3b395569235e1a7d
Jul 13 20:17:58.957343 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:58.957282    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 20:17:59.494005 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:17:59.493967    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-dc6dd9667-pq4h9" event={"ID":"8b63f5d1-ddc1-4b49-9bd1-69133ae2919f","Type":"ContainerStarted","Data":"1d90c95ff7eaf17cd5398813b23c446087c11a609e3d01ac3b395569235e1a7d"}
Jul 13 20:18:00.500542 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:00.500501    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-dc6dd9667-pq4h9" event={"ID":"8b63f5d1-ddc1-4b49-9bd1-69133ae2919f","Type":"ContainerStarted","Data":"2bcae2b04d4cc4b36689fe3433539f1b09d388562964eae7548567c935e12c25"}
Jul 13 20:18:00.500986 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:00.500630    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-dc6dd9667-pq4h9"
Jul 13 20:18:00.520899 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:00.520817    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-dc6dd9667-pq4h9" podStartSLOduration=1.973806192 podStartE2EDuration="2.520795492s" totalImagesPullingTime="546.9893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:17:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:18:00.518037676 +0000 UTC m=+5302.031836856" watchObservedRunningTime="2026-07-13 20:18:00.520795492 +0000 UTC m=+5302.034594654"
Jul 13 20:18:31.508370 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:31.508335    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-dc6dd9667-pq4h9"
Jul 13 20:18:31.559914 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:31.559882    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-bdf85754c-vt4bs"]
Jul 13 20:18:31.560164 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:31.560116    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-bdf85754c-vt4bs" podUID="11d53207-54ca-4ffd-a9ed-623f2e7555ce" containerName="manager" containerID="cri-o://439e163cd1e95fcb3042d88de5ee5ffc876843e47eaf3eb9f79d782d970c77e2" gracePeriod=30
Jul 13 20:18:32.112732 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:32.112704    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-bdf85754c-vt4bs"
Jul 13 20:18:32.243393 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:32.243284    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11d53207-54ca-4ffd-a9ed-623f2e7555ce-cert\") pod \"11d53207-54ca-4ffd-a9ed-623f2e7555ce\" (UID: \"11d53207-54ca-4ffd-a9ed-623f2e7555ce\") "
Jul 13 20:18:32.243576 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:32.243449    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kz6sb\" (UniqueName: \"kubernetes.io/projected/11d53207-54ca-4ffd-a9ed-623f2e7555ce-kube-api-access-kz6sb\") pod \"11d53207-54ca-4ffd-a9ed-623f2e7555ce\" (UID: \"11d53207-54ca-4ffd-a9ed-623f2e7555ce\") "
Jul 13 20:18:32.245739 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:32.245708    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11d53207-54ca-4ffd-a9ed-623f2e7555ce-cert" (OuterVolumeSpecName: "cert") pod "11d53207-54ca-4ffd-a9ed-623f2e7555ce" (UID: "11d53207-54ca-4ffd-a9ed-623f2e7555ce"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:18:32.245739 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:32.245726    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d53207-54ca-4ffd-a9ed-623f2e7555ce-kube-api-access-kz6sb" (OuterVolumeSpecName: "kube-api-access-kz6sb") pod "11d53207-54ca-4ffd-a9ed-623f2e7555ce" (UID: "11d53207-54ca-4ffd-a9ed-623f2e7555ce"). InnerVolumeSpecName "kube-api-access-kz6sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:18:32.344250 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:32.344210    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz6sb\" (UniqueName: \"kubernetes.io/projected/11d53207-54ca-4ffd-a9ed-623f2e7555ce-kube-api-access-kz6sb\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:18:32.344250 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:32.344243    2569 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11d53207-54ca-4ffd-a9ed-623f2e7555ce-cert\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:18:32.652430 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:32.652393    2569 generic.go:356] "Generic (PLEG): container finished" podID="11d53207-54ca-4ffd-a9ed-623f2e7555ce" containerID="439e163cd1e95fcb3042d88de5ee5ffc876843e47eaf3eb9f79d782d970c77e2" exitCode=0
Jul 13 20:18:32.652905 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:32.652449    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-bdf85754c-vt4bs"
Jul 13 20:18:32.652905 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:32.652474    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-bdf85754c-vt4bs" event={"ID":"11d53207-54ca-4ffd-a9ed-623f2e7555ce","Type":"ContainerDied","Data":"439e163cd1e95fcb3042d88de5ee5ffc876843e47eaf3eb9f79d782d970c77e2"}
Jul 13 20:18:32.652905 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:32.652527    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-bdf85754c-vt4bs" event={"ID":"11d53207-54ca-4ffd-a9ed-623f2e7555ce","Type":"ContainerDied","Data":"a168669d61492004ff7a5a198d75aea8484ebafd748536dc8a48923c06f6eb42"}
Jul 13 20:18:32.652905 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:32.652550    2569 scope.go:117] "RemoveContainer" containerID="439e163cd1e95fcb3042d88de5ee5ffc876843e47eaf3eb9f79d782d970c77e2"
Jul 13 20:18:32.663981 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:32.663962    2569 scope.go:117] "RemoveContainer" containerID="439e163cd1e95fcb3042d88de5ee5ffc876843e47eaf3eb9f79d782d970c77e2"
Jul 13 20:18:32.664330 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:18:32.664306    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"439e163cd1e95fcb3042d88de5ee5ffc876843e47eaf3eb9f79d782d970c77e2\": container with ID starting with 439e163cd1e95fcb3042d88de5ee5ffc876843e47eaf3eb9f79d782d970c77e2 not found: ID does not exist" containerID="439e163cd1e95fcb3042d88de5ee5ffc876843e47eaf3eb9f79d782d970c77e2"
Jul 13 20:18:32.664397 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:32.664342    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"439e163cd1e95fcb3042d88de5ee5ffc876843e47eaf3eb9f79d782d970c77e2"} err="failed to get container status \"439e163cd1e95fcb3042d88de5ee5ffc876843e47eaf3eb9f79d782d970c77e2\": rpc error: code = NotFound desc = could not find container \"439e163cd1e95fcb3042d88de5ee5ffc876843e47eaf3eb9f79d782d970c77e2\": container with ID starting with 439e163cd1e95fcb3042d88de5ee5ffc876843e47eaf3eb9f79d782d970c77e2 not found: ID does not exist"
Jul 13 20:18:32.674414 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:32.674383    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-bdf85754c-vt4bs"]
Jul 13 20:18:32.679306 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:32.679270    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-bdf85754c-vt4bs"]
Jul 13 20:18:33.060370 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:18:33.060271    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11d53207-54ca-4ffd-a9ed-623f2e7555ce" path="/var/lib/kubelet/pods/11d53207-54ca-4ffd-a9ed-623f2e7555ce/volumes"
Jul 13 20:19:39.413838 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:19:39.413729    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 20:19:39.418917 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:19:39.418237    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 20:19:39.423879 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:19:39.423861    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 20:19:39.427907 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:19:39.427881    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 20:22:55.547739 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:22:55.547694    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-stwlp/must-gather-wkpst"]
Jul 13 20:22:55.548385 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:22:55.548163    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d53207-54ca-4ffd-a9ed-623f2e7555ce" containerName="manager"
Jul 13 20:22:55.548385 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:22:55.548180    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d53207-54ca-4ffd-a9ed-623f2e7555ce" containerName="manager"
Jul 13 20:22:55.548385 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:22:55.548310    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="11d53207-54ca-4ffd-a9ed-623f2e7555ce" containerName="manager"
Jul 13 20:22:55.551516 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:22:55.551493    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-stwlp/must-gather-wkpst"
Jul 13 20:22:55.554630 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:22:55.554606    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-stwlp\"/\"kube-root-ca.crt\""
Jul 13 20:22:55.554930 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:22:55.554911    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-stwlp\"/\"openshift-service-ca.crt\""
Jul 13 20:22:55.563319 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:22:55.563268    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-stwlp/must-gather-wkpst"]
Jul 13 20:22:55.634347 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:22:55.634309    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn7d7\" (UniqueName: \"kubernetes.io/projected/a25b398a-0ead-43dd-aa75-dc9acff01843-kube-api-access-gn7d7\") pod \"must-gather-wkpst\" (UID: \"a25b398a-0ead-43dd-aa75-dc9acff01843\") " pod="openshift-must-gather-stwlp/must-gather-wkpst"
Jul 13 20:22:55.634517 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:22:55.634357    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a25b398a-0ead-43dd-aa75-dc9acff01843-must-gather-output\") pod \"must-gather-wkpst\" (UID: \"a25b398a-0ead-43dd-aa75-dc9acff01843\") " pod="openshift-must-gather-stwlp/must-gather-wkpst"
Jul 13 20:22:55.735176 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:22:55.735136    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn7d7\" (UniqueName: \"kubernetes.io/projected/a25b398a-0ead-43dd-aa75-dc9acff01843-kube-api-access-gn7d7\") pod \"must-gather-wkpst\" (UID: \"a25b398a-0ead-43dd-aa75-dc9acff01843\") " pod="openshift-must-gather-stwlp/must-gather-wkpst"
Jul 13 20:22:55.735402 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:22:55.735198    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a25b398a-0ead-43dd-aa75-dc9acff01843-must-gather-output\") pod \"must-gather-wkpst\" (UID: \"a25b398a-0ead-43dd-aa75-dc9acff01843\") " pod="openshift-must-gather-stwlp/must-gather-wkpst"
Jul 13 20:22:55.735600 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:22:55.735575    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a25b398a-0ead-43dd-aa75-dc9acff01843-must-gather-output\") pod \"must-gather-wkpst\" (UID: \"a25b398a-0ead-43dd-aa75-dc9acff01843\") " pod="openshift-must-gather-stwlp/must-gather-wkpst"
Jul 13 20:22:55.743623 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:22:55.743601    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn7d7\" (UniqueName: \"kubernetes.io/projected/a25b398a-0ead-43dd-aa75-dc9acff01843-kube-api-access-gn7d7\") pod \"must-gather-wkpst\" (UID: \"a25b398a-0ead-43dd-aa75-dc9acff01843\") " pod="openshift-must-gather-stwlp/must-gather-wkpst"
Jul 13 20:22:55.861592 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:22:55.861557    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-stwlp/must-gather-wkpst"
Jul 13 20:22:55.997072 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:22:55.997046    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-stwlp/must-gather-wkpst"]
Jul 13 20:22:55.998965 ip-10-0-139-134 kubenswrapper[2569]: W0713 20:22:55.998940    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda25b398a_0ead_43dd_aa75_dc9acff01843.slice/crio-4823516467ad09e23e3202104699968308b2e63cbb1d8656aca1e216d553bee5 WatchSource:0}: Error finding container 4823516467ad09e23e3202104699968308b2e63cbb1d8656aca1e216d553bee5: Status 404 returned error can't find the container with id 4823516467ad09e23e3202104699968308b2e63cbb1d8656aca1e216d553bee5
Jul 13 20:22:56.856590 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:22:56.856543    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-stwlp/must-gather-wkpst" event={"ID":"a25b398a-0ead-43dd-aa75-dc9acff01843","Type":"ContainerStarted","Data":"4823516467ad09e23e3202104699968308b2e63cbb1d8656aca1e216d553bee5"}
Jul 13 20:23:02.890704 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:02.890617    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-stwlp/must-gather-wkpst" event={"ID":"a25b398a-0ead-43dd-aa75-dc9acff01843","Type":"ContainerStarted","Data":"5b76abdd5aeb5f62088905a6b618fc47ef652ed07ef914b44a6f1dcd0ef5eed0"}
Jul 13 20:23:02.890704 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:02.890669    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-stwlp/must-gather-wkpst" event={"ID":"a25b398a-0ead-43dd-aa75-dc9acff01843","Type":"ContainerStarted","Data":"d19d472c98ca6d174311ddf026cc50aa3dc79846b5513797123780890a1fbec8"}
Jul 13 20:23:02.912584 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:02.912523    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-stwlp/must-gather-wkpst" podStartSLOduration=1.533921303 podStartE2EDuration="7.912506s" totalImagesPullingTime="6.378584697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:23:02.910246932 +0000 UTC m=+5604.424046099" watchObservedRunningTime="2026-07-13 20:23:02.912506 +0000 UTC m=+5604.426305384"
Jul 13 20:23:12.172668 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:12.172615    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2_2ce120d6-0e30-45b7-90fc-f7f76654786e/main/0.log"
Jul 13 20:23:12.227524 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:12.227462    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff_42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446/main/0.log"
Jul 13 20:23:14.144575 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:14.144530    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2_2ce120d6-0e30-45b7-90fc-f7f76654786e/main/0.log"
Jul 13 20:23:14.174077 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:14.174042    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff_42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446/main/0.log"
Jul 13 20:23:16.036024 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:16.035993    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2_2ce120d6-0e30-45b7-90fc-f7f76654786e/main/0.log"
Jul 13 20:23:16.076522 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:16.076483    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff_42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446/main/0.log"
Jul 13 20:23:18.031697 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:18.031658    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2_2ce120d6-0e30-45b7-90fc-f7f76654786e/main/0.log"
Jul 13 20:23:18.049858 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:18.049822    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff_42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446/main/0.log"
Jul 13 20:23:19.840187 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:19.840155    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2_2ce120d6-0e30-45b7-90fc-f7f76654786e/main/0.log"
Jul 13 20:23:19.857396 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:19.857364    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff_42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446/main/0.log"
Jul 13 20:23:21.700195 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:21.700152    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2_2ce120d6-0e30-45b7-90fc-f7f76654786e/main/0.log"
Jul 13 20:23:21.718306 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:21.718259    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff_42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446/main/0.log"
Jul 13 20:23:23.409565 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:23.409526    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2_2ce120d6-0e30-45b7-90fc-f7f76654786e/main/0.log"
Jul 13 20:23:23.452172 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:23.452130    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff_42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446/main/0.log"
Jul 13 20:23:25.321011 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:25.320972    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2_2ce120d6-0e30-45b7-90fc-f7f76654786e/main/0.log"
Jul 13 20:23:25.337427 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:25.337396    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff_42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446/main/0.log"
Jul 13 20:23:27.055102 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:27.055059    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2_2ce120d6-0e30-45b7-90fc-f7f76654786e/main/0.log"
Jul 13 20:23:27.070170 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:27.070141    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff_42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446/main/0.log"
Jul 13 20:23:28.742874 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:28.742841    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2_2ce120d6-0e30-45b7-90fc-f7f76654786e/main/0.log"
Jul 13 20:23:28.764261 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:28.764228    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff_42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446/main/0.log"
Jul 13 20:23:30.505742 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:30.505701    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2_2ce120d6-0e30-45b7-90fc-f7f76654786e/main/0.log"
Jul 13 20:23:30.528427 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:30.528396    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff_42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446/main/0.log"
Jul 13 20:23:32.259686 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:32.259643    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2_2ce120d6-0e30-45b7-90fc-f7f76654786e/main/0.log"
Jul 13 20:23:32.280158 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:32.280132    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff_42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446/main/0.log"
Jul 13 20:23:34.026542 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:34.026508    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2_2ce120d6-0e30-45b7-90fc-f7f76654786e/main/0.log"
Jul 13 20:23:34.042220 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:34.042193    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff_42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446/main/0.log"
Jul 13 20:23:35.688830 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:35.688796    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2_2ce120d6-0e30-45b7-90fc-f7f76654786e/main/0.log"
Jul 13 20:23:35.703238 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:35.703212    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff_42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446/main/0.log"
Jul 13 20:23:37.466833 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:37.466791    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2_2ce120d6-0e30-45b7-90fc-f7f76654786e/main/0.log"
Jul 13 20:23:37.490322 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:37.490275    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff_42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446/main/0.log"
Jul 13 20:23:39.209204 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:39.209171    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2_2ce120d6-0e30-45b7-90fc-f7f76654786e/main/0.log"
Jul 13 20:23:39.224527 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:39.224489    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff_42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446/main/0.log"
Jul 13 20:23:41.538405 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:41.538371    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-586b57644b-q24lm_4fcfa1f0-d0ea-488a-a414-d09b8085aae5/router/0.log"
Jul 13 20:23:43.175352 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:43.175327    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-586b57644b-q24lm_4fcfa1f0-d0ea-488a-a414-d09b8085aae5/router/0.log"
Jul 13 20:23:44.880405 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:44.880373    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-586b57644b-q24lm_4fcfa1f0-d0ea-488a-a414-d09b8085aae5/router/0.log"
Jul 13 20:23:46.476657 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:46.476632    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-586b57644b-q24lm_4fcfa1f0-d0ea-488a-a414-d09b8085aae5/router/0.log"
Jul 13 20:23:47.500188 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:47.500146    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:23:47.522272 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:47.522227    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:23:48.988962 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:48.988923    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:23:49.004408 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:49.004372    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:23:50.494652 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:50.494617    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:23:50.512816 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:50.512789    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:23:51.912395 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:51.912357    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:23:51.928205 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:51.928170    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:23:53.353737 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:53.353698    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:23:53.374997 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:53.374970    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:23:54.790243 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:54.790209    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:23:54.813449 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:54.813412    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:23:56.181258 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:56.181210    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:23:56.246866 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:56.246837    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:23:57.682474 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:57.682431    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:23:57.708724 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:57.708687    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:23:59.138065 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:59.138025    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:23:59.176255 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:23:59.176226    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:00.580625 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:00.580576    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:00.600645 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:00.600595    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:02.022179 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:02.022144    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:02.058568 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:02.058536    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:03.501012 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:03.500981    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:03.521557 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:03.521522    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:05.009195 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:05.009154    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:05.050234 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:05.050200    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:06.634402 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:06.634366    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:06.670533 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:06.670503    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:08.074726 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:08.074690    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:08.092134 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:08.092104    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:09.526266 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:09.526233    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:09.551253 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:09.551221    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:11.050521 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:11.050483    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:11.105758 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:11.105730    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:12.516031 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:12.515991    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:12.588841 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:12.588809    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:14.075452 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:14.075421    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:14.105310 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:14.105264    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:15.644779 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:15.644746    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:15.667238 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:15.667197    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:17.080900 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:17.080860    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:17.136759 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:17.136726    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:18.707725 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:18.707684    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:18.726894 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:18.726863    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:20.139249 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:20.139215    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:20.155757 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:20.155705    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:21.623473 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:21.623443    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:21.679128 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:21.679059    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:23.346431 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:23.346399    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:23.409693 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:23.409654    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:24.900512 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:24.900463    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:24.985444 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:24.985404    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:26.502544 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:26.502511    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:26.530648 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:26.530616    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:27.977413 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:27.977370    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:27.995093 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:27.995061    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:29.451549 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:29.451513    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:29.507053 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:29.507024    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:31.017799 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:31.017764    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:31.045394 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:31.045351    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:32.663619 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:32.663581    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:32.786256 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:32.786228    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:34.358787 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:34.358751    2569 generic.go:356] "Generic (PLEG): container finished" podID="a25b398a-0ead-43dd-aa75-dc9acff01843" containerID="d19d472c98ca6d174311ddf026cc50aa3dc79846b5513797123780890a1fbec8" exitCode=0
Jul 13 20:24:34.359353 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:34.358825    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-stwlp/must-gather-wkpst" event={"ID":"a25b398a-0ead-43dd-aa75-dc9acff01843","Type":"ContainerDied","Data":"d19d472c98ca6d174311ddf026cc50aa3dc79846b5513797123780890a1fbec8"}
Jul 13 20:24:34.359353 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:34.359246    2569 scope.go:117] "RemoveContainer" containerID="d19d472c98ca6d174311ddf026cc50aa3dc79846b5513797123780890a1fbec8"
Jul 13 20:24:35.109340 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:35.109309    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-stwlp_must-gather-wkpst_a25b398a-0ead-43dd-aa75-dc9acff01843/gather/0.log"
Jul 13 20:24:36.094014 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:36.093973    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s6wcw/must-gather-b2w4x"]
Jul 13 20:24:36.098803 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:36.098777    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s6wcw/must-gather-b2w4x"
Jul 13 20:24:36.105518 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:36.105495    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s6wcw\"/\"default-dockercfg-g9b4n\""
Jul 13 20:24:36.106151 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:36.106133    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s6wcw\"/\"openshift-service-ca.crt\""
Jul 13 20:24:36.106258 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:36.106159    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s6wcw\"/\"kube-root-ca.crt\""
Jul 13 20:24:36.129823 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:36.129788    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s6wcw/must-gather-b2w4x"]
Jul 13 20:24:36.250620 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:36.250581    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2e5e37b8-013a-4f43-842c-e5c16edc9352-must-gather-output\") pod \"must-gather-b2w4x\" (UID: \"2e5e37b8-013a-4f43-842c-e5c16edc9352\") " pod="openshift-must-gather-s6wcw/must-gather-b2w4x"
Jul 13 20:24:36.250814 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:36.250630    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px46x\" (UniqueName: \"kubernetes.io/projected/2e5e37b8-013a-4f43-842c-e5c16edc9352-kube-api-access-px46x\") pod \"must-gather-b2w4x\" (UID: \"2e5e37b8-013a-4f43-842c-e5c16edc9352\") " pod="openshift-must-gather-s6wcw/must-gather-b2w4x"
Jul 13 20:24:36.352190 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:36.352088    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2e5e37b8-013a-4f43-842c-e5c16edc9352-must-gather-output\") pod \"must-gather-b2w4x\" (UID: \"2e5e37b8-013a-4f43-842c-e5c16edc9352\") " pod="openshift-must-gather-s6wcw/must-gather-b2w4x"
Jul 13 20:24:36.352190 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:36.352133    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px46x\" (UniqueName: \"kubernetes.io/projected/2e5e37b8-013a-4f43-842c-e5c16edc9352-kube-api-access-px46x\") pod \"must-gather-b2w4x\" (UID: \"2e5e37b8-013a-4f43-842c-e5c16edc9352\") " pod="openshift-must-gather-s6wcw/must-gather-b2w4x"
Jul 13 20:24:36.352577 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:36.352554    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2e5e37b8-013a-4f43-842c-e5c16edc9352-must-gather-output\") pod \"must-gather-b2w4x\" (UID: \"2e5e37b8-013a-4f43-842c-e5c16edc9352\") " pod="openshift-must-gather-s6wcw/must-gather-b2w4x"
Jul 13 20:24:36.380564 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:36.380528    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px46x\" (UniqueName: \"kubernetes.io/projected/2e5e37b8-013a-4f43-842c-e5c16edc9352-kube-api-access-px46x\") pod \"must-gather-b2w4x\" (UID: \"2e5e37b8-013a-4f43-842c-e5c16edc9352\") " pod="openshift-must-gather-s6wcw/must-gather-b2w4x"
Jul 13 20:24:36.407653 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:36.407618    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s6wcw/must-gather-b2w4x"
Jul 13 20:24:36.592897 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:36.592872    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s6wcw/must-gather-b2w4x"]
Jul 13 20:24:36.595830 ip-10-0-139-134 kubenswrapper[2569]: W0713 20:24:36.595803    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e5e37b8_013a_4f43_842c_e5c16edc9352.slice/crio-9cb76b55254b3ddc64aaf2f7505ef1783b9a43749fbd8d5a6535514decd159e7 WatchSource:0}: Error finding container 9cb76b55254b3ddc64aaf2f7505ef1783b9a43749fbd8d5a6535514decd159e7: Status 404 returned error can't find the container with id 9cb76b55254b3ddc64aaf2f7505ef1783b9a43749fbd8d5a6535514decd159e7
Jul 13 20:24:36.597808 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:36.597790    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 20:24:37.371996 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:37.371964    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s6wcw/must-gather-b2w4x" event={"ID":"2e5e37b8-013a-4f43-842c-e5c16edc9352","Type":"ContainerStarted","Data":"9cb76b55254b3ddc64aaf2f7505ef1783b9a43749fbd8d5a6535514decd159e7"}
Jul 13 20:24:38.380255 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:38.380221    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s6wcw/must-gather-b2w4x" event={"ID":"2e5e37b8-013a-4f43-842c-e5c16edc9352","Type":"ContainerStarted","Data":"d69e614394629fc0eb9a28ca31be9c73f083c5f96e8107e6dcc065809972acf3"}
Jul 13 20:24:38.380255 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:38.380263    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s6wcw/must-gather-b2w4x" event={"ID":"2e5e37b8-013a-4f43-842c-e5c16edc9352","Type":"ContainerStarted","Data":"6759118c2b57253efafa423c181cdad02dce89eca07c7945ae43d10fda3a0e14"}
Jul 13 20:24:39.452859 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:39.451475    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 20:24:39.470568 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:39.457557    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 20:24:39.470568 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:39.464324    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 20:24:39.471478 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:39.471457    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v76kh_d254ad6f-c371-4b41-a97d-97056891fdf6/ovn-acl-logging/0.log"
Jul 13 20:24:39.562344 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:39.561986    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-66pbz_b1f63a2f-aa86-415d-8c72-78c96c560625/global-pull-secret-syncer/0.log"
Jul 13 20:24:40.074605 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:40.074567    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-z9pwv_eacba2f0-6acf-4ce6-a960-0f5b0c8a77d9/konnectivity-agent/0.log"
Jul 13 20:24:40.245647 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:40.245610    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-134.ec2.internal_6d1a4d271968fb37c56f11e30e3e00f8/haproxy/0.log"
Jul 13 20:24:40.872986 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:40.872897    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s6wcw/must-gather-b2w4x" podStartSLOduration=4.102322755 podStartE2EDuration="4.872873959s" totalImagesPullingTime="770.551204ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:24:38.442776124 +0000 UTC m=+5699.956575290" watchObservedRunningTime="2026-07-13 20:24:40.872873959 +0000 UTC m=+5702.386673103"
Jul 13 20:24:40.873506 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:40.873186    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-stwlp/must-gather-wkpst"]
Jul 13 20:24:40.873753 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:40.873478    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-stwlp/must-gather-wkpst" podUID="a25b398a-0ead-43dd-aa75-dc9acff01843" containerName="copy" containerID="cri-o://5b76abdd5aeb5f62088905a6b618fc47ef652ed07ef914b44a6f1dcd0ef5eed0" gracePeriod=2
Jul 13 20:24:40.878310 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:24:40.877129    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-wkpst\" is forbidden: User \"system:node:ip-10-0-139-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-stwlp\": no relationship found between node 'ip-10-0-139-134.ec2.internal' and this object" podUID="a25b398a-0ead-43dd-aa75-dc9acff01843" pod="openshift-must-gather-stwlp/must-gather-wkpst"
Jul 13 20:24:40.880977 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:40.880947    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-stwlp/must-gather-wkpst"]
Jul 13 20:24:41.267625 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:41.267598    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-stwlp_must-gather-wkpst_a25b398a-0ead-43dd-aa75-dc9acff01843/copy/0.log"
Jul 13 20:24:41.281510 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:41.281474    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-stwlp/must-gather-wkpst"
Jul 13 20:24:41.397986 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:41.397906    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn7d7\" (UniqueName: \"kubernetes.io/projected/a25b398a-0ead-43dd-aa75-dc9acff01843-kube-api-access-gn7d7\") pod \"a25b398a-0ead-43dd-aa75-dc9acff01843\" (UID: \"a25b398a-0ead-43dd-aa75-dc9acff01843\") "
Jul 13 20:24:41.398284 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:41.398268    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a25b398a-0ead-43dd-aa75-dc9acff01843-must-gather-output\") pod \"a25b398a-0ead-43dd-aa75-dc9acff01843\" (UID: \"a25b398a-0ead-43dd-aa75-dc9acff01843\") "
Jul 13 20:24:41.411823 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:41.411778    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25b398a-0ead-43dd-aa75-dc9acff01843-kube-api-access-gn7d7" (OuterVolumeSpecName: "kube-api-access-gn7d7") pod "a25b398a-0ead-43dd-aa75-dc9acff01843" (UID: "a25b398a-0ead-43dd-aa75-dc9acff01843"). InnerVolumeSpecName "kube-api-access-gn7d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:24:41.424128 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:41.423748    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25b398a-0ead-43dd-aa75-dc9acff01843-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "a25b398a-0ead-43dd-aa75-dc9acff01843" (UID: "a25b398a-0ead-43dd-aa75-dc9acff01843"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:24:41.424128 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:41.424049    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-stwlp_must-gather-wkpst_a25b398a-0ead-43dd-aa75-dc9acff01843/copy/0.log"
Jul 13 20:24:41.425323 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:41.424671    2569 generic.go:356] "Generic (PLEG): container finished" podID="a25b398a-0ead-43dd-aa75-dc9acff01843" containerID="5b76abdd5aeb5f62088905a6b618fc47ef652ed07ef914b44a6f1dcd0ef5eed0" exitCode=143
Jul 13 20:24:41.425323 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:41.424803    2569 scope.go:117] "RemoveContainer" containerID="5b76abdd5aeb5f62088905a6b618fc47ef652ed07ef914b44a6f1dcd0ef5eed0"
Jul 13 20:24:41.425323 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:41.424985    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-stwlp/must-gather-wkpst"
Jul 13 20:24:41.447115 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:41.447051    2569 scope.go:117] "RemoveContainer" containerID="d19d472c98ca6d174311ddf026cc50aa3dc79846b5513797123780890a1fbec8"
Jul 13 20:24:41.470218 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:41.470184    2569 scope.go:117] "RemoveContainer" containerID="5b76abdd5aeb5f62088905a6b618fc47ef652ed07ef914b44a6f1dcd0ef5eed0"
Jul 13 20:24:41.470805 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:24:41.470776    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b76abdd5aeb5f62088905a6b618fc47ef652ed07ef914b44a6f1dcd0ef5eed0\": container with ID starting with 5b76abdd5aeb5f62088905a6b618fc47ef652ed07ef914b44a6f1dcd0ef5eed0 not found: ID does not exist" containerID="5b76abdd5aeb5f62088905a6b618fc47ef652ed07ef914b44a6f1dcd0ef5eed0"
Jul 13 20:24:41.470955 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:41.470928    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b76abdd5aeb5f62088905a6b618fc47ef652ed07ef914b44a6f1dcd0ef5eed0"} err="failed to get container status \"5b76abdd5aeb5f62088905a6b618fc47ef652ed07ef914b44a6f1dcd0ef5eed0\": rpc error: code = NotFound desc = could not find container \"5b76abdd5aeb5f62088905a6b618fc47ef652ed07ef914b44a6f1dcd0ef5eed0\": container with ID starting with 5b76abdd5aeb5f62088905a6b618fc47ef652ed07ef914b44a6f1dcd0ef5eed0 not found: ID does not exist"
Jul 13 20:24:41.471071 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:41.471059    2569 scope.go:117] "RemoveContainer" containerID="d19d472c98ca6d174311ddf026cc50aa3dc79846b5513797123780890a1fbec8"
Jul 13 20:24:41.471456 ip-10-0-139-134 kubenswrapper[2569]: E0713 20:24:41.471436    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d19d472c98ca6d174311ddf026cc50aa3dc79846b5513797123780890a1fbec8\": container with ID starting with d19d472c98ca6d174311ddf026cc50aa3dc79846b5513797123780890a1fbec8 not found: ID does not exist" containerID="d19d472c98ca6d174311ddf026cc50aa3dc79846b5513797123780890a1fbec8"
Jul 13 20:24:41.471603 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:41.471581    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d19d472c98ca6d174311ddf026cc50aa3dc79846b5513797123780890a1fbec8"} err="failed to get container status \"d19d472c98ca6d174311ddf026cc50aa3dc79846b5513797123780890a1fbec8\": rpc error: code = NotFound desc = could not find container \"d19d472c98ca6d174311ddf026cc50aa3dc79846b5513797123780890a1fbec8\": container with ID starting with d19d472c98ca6d174311ddf026cc50aa3dc79846b5513797123780890a1fbec8 not found: ID does not exist"
Jul 13 20:24:41.499409 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:41.499376    2569 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a25b398a-0ead-43dd-aa75-dc9acff01843-must-gather-output\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:24:41.499672 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:41.499656    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn7d7\" (UniqueName: \"kubernetes.io/projected/a25b398a-0ead-43dd-aa75-dc9acff01843-kube-api-access-gn7d7\") on node \"ip-10-0-139-134.ec2.internal\" DevicePath \"\""
Jul 13 20:24:43.063837 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:43.063803    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a25b398a-0ead-43dd-aa75-dc9acff01843" path="/var/lib/kubelet/pods/a25b398a-0ead-43dd-aa75-dc9acff01843/volumes"
Jul 13 20:24:45.064840 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:45.064764    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-cfthr_0ea71281-4a8f-4c25-b9ef-4daa6f7465ce/authorino/0.log"
Jul 13 20:24:45.118486 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:45.118446    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-xdn96_eeaf5dd2-2dd3-48fe-9953-305cb53e5440/manager/0.log"
Jul 13 20:24:48.013978 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:48.013747    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-8lv7g_42789374-5d5e-4463-9db7-50101ecf2a58/kube-state-metrics/0.log"
Jul 13 20:24:48.081567 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:48.081522    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-8lv7g_42789374-5d5e-4463-9db7-50101ecf2a58/kube-rbac-proxy-main/0.log"
Jul 13 20:24:48.157857 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:48.157822    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-8lv7g_42789374-5d5e-4463-9db7-50101ecf2a58/kube-rbac-proxy-self/0.log"
Jul 13 20:24:48.255730 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:48.255700    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-ffb8ff984-rft4x_d5e724d7-11cf-4346-8295-ef996b0eb651/metrics-server/0.log"
Jul 13 20:24:48.347385 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:48.347351    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cw9bv_9fcc6e8d-3e22-4ea0-955c-c631bbcbd879/node-exporter/0.log"
Jul 13 20:24:48.382448 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:48.382424    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cw9bv_9fcc6e8d-3e22-4ea0-955c-c631bbcbd879/kube-rbac-proxy/0.log"
Jul 13 20:24:48.418926 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:48.418901    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cw9bv_9fcc6e8d-3e22-4ea0-955c-c631bbcbd879/init-textfile/0.log"
Jul 13 20:24:48.729776 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:48.729678    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-dmh6c_38462958-ea11-4e61-aad7-6154f7fc2fa8/kube-rbac-proxy-main/0.log"
Jul 13 20:24:48.776416 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:48.776374    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-dmh6c_38462958-ea11-4e61-aad7-6154f7fc2fa8/kube-rbac-proxy-self/0.log"
Jul 13 20:24:48.827935 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:48.827891    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-dmh6c_38462958-ea11-4e61-aad7-6154f7fc2fa8/openshift-state-metrics/0.log"
Jul 13 20:24:48.912646 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:48.912601    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a4f3c1eb-933f-44b3-9461-ec3da50e38b6/prometheus/0.log"
Jul 13 20:24:48.973543 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:48.973508    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a4f3c1eb-933f-44b3-9461-ec3da50e38b6/config-reloader/0.log"
Jul 13 20:24:49.025314 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.025210    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a4f3c1eb-933f-44b3-9461-ec3da50e38b6/thanos-sidecar/0.log"
Jul 13 20:24:49.075437 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.075407    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a4f3c1eb-933f-44b3-9461-ec3da50e38b6/kube-rbac-proxy-web/0.log"
Jul 13 20:24:49.131000 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.130971    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a4f3c1eb-933f-44b3-9461-ec3da50e38b6/kube-rbac-proxy/0.log"
Jul 13 20:24:49.172954 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.172926    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a4f3c1eb-933f-44b3-9461-ec3da50e38b6/kube-rbac-proxy-thanos/0.log"
Jul 13 20:24:49.234943 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.234911    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a4f3c1eb-933f-44b3-9461-ec3da50e38b6/init-config-reloader/0.log"
Jul 13 20:24:49.455631 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.455588    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-r78ll_25d90578-2923-49f9-88e8-cf9f72a0ebb1/prometheus-operator-admission-webhook/0.log"
Jul 13 20:24:49.615689 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.615654    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8"]
Jul 13 20:24:49.616084 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.616066    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a25b398a-0ead-43dd-aa75-dc9acff01843" containerName="copy"
Jul 13 20:24:49.616175 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.616087    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a25b398a-0ead-43dd-aa75-dc9acff01843" containerName="copy"
Jul 13 20:24:49.616175 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.616122    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a25b398a-0ead-43dd-aa75-dc9acff01843" containerName="gather"
Jul 13 20:24:49.616175 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.616131    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a25b398a-0ead-43dd-aa75-dc9acff01843" containerName="gather"
Jul 13 20:24:49.616351 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.616215    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a25b398a-0ead-43dd-aa75-dc9acff01843" containerName="copy"
Jul 13 20:24:49.616351 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.616227    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a25b398a-0ead-43dd-aa75-dc9acff01843" containerName="gather"
Jul 13 20:24:49.620711 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.620685    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8"
Jul 13 20:24:49.657437 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.657400    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8"]
Jul 13 20:24:49.705112 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.705073    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8f0d9b47-0865-402c-bfd5-a95e85236c27-proc\") pod \"perf-node-gather-daemonset-gtnw8\" (UID: \"8f0d9b47-0865-402c-bfd5-a95e85236c27\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8"
Jul 13 20:24:49.705112 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.705115    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f0d9b47-0865-402c-bfd5-a95e85236c27-lib-modules\") pod \"perf-node-gather-daemonset-gtnw8\" (UID: \"8f0d9b47-0865-402c-bfd5-a95e85236c27\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8"
Jul 13 20:24:49.705361 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.705138    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8f0d9b47-0865-402c-bfd5-a95e85236c27-podres\") pod \"perf-node-gather-daemonset-gtnw8\" (UID: \"8f0d9b47-0865-402c-bfd5-a95e85236c27\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8"
Jul 13 20:24:49.705361 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.705224    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f0d9b47-0865-402c-bfd5-a95e85236c27-sys\") pod \"perf-node-gather-daemonset-gtnw8\" (UID: \"8f0d9b47-0865-402c-bfd5-a95e85236c27\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8"
Jul 13 20:24:49.705361 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.705309    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf56k\" (UniqueName: \"kubernetes.io/projected/8f0d9b47-0865-402c-bfd5-a95e85236c27-kube-api-access-pf56k\") pod \"perf-node-gather-daemonset-gtnw8\" (UID: \"8f0d9b47-0865-402c-bfd5-a95e85236c27\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8"
Jul 13 20:24:49.806453 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.806375    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f0d9b47-0865-402c-bfd5-a95e85236c27-sys\") pod \"perf-node-gather-daemonset-gtnw8\" (UID: \"8f0d9b47-0865-402c-bfd5-a95e85236c27\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8"
Jul 13 20:24:49.806453 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.806437    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf56k\" (UniqueName: \"kubernetes.io/projected/8f0d9b47-0865-402c-bfd5-a95e85236c27-kube-api-access-pf56k\") pod \"perf-node-gather-daemonset-gtnw8\" (UID: \"8f0d9b47-0865-402c-bfd5-a95e85236c27\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8"
Jul 13 20:24:49.806654 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.806486    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8f0d9b47-0865-402c-bfd5-a95e85236c27-proc\") pod \"perf-node-gather-daemonset-gtnw8\" (UID: \"8f0d9b47-0865-402c-bfd5-a95e85236c27\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8"
Jul 13 20:24:49.806654 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.806502    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f0d9b47-0865-402c-bfd5-a95e85236c27-lib-modules\") pod \"perf-node-gather-daemonset-gtnw8\" (UID: \"8f0d9b47-0865-402c-bfd5-a95e85236c27\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8"
Jul 13 20:24:49.806654 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.806527    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8f0d9b47-0865-402c-bfd5-a95e85236c27-podres\") pod \"perf-node-gather-daemonset-gtnw8\" (UID: \"8f0d9b47-0865-402c-bfd5-a95e85236c27\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8"
Jul 13 20:24:49.806654 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.806534    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f0d9b47-0865-402c-bfd5-a95e85236c27-sys\") pod \"perf-node-gather-daemonset-gtnw8\" (UID: \"8f0d9b47-0865-402c-bfd5-a95e85236c27\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8"
Jul 13 20:24:49.806654 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.806590    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8f0d9b47-0865-402c-bfd5-a95e85236c27-proc\") pod \"perf-node-gather-daemonset-gtnw8\" (UID: \"8f0d9b47-0865-402c-bfd5-a95e85236c27\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8"
Jul 13 20:24:49.806816 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.806666    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8f0d9b47-0865-402c-bfd5-a95e85236c27-podres\") pod \"perf-node-gather-daemonset-gtnw8\" (UID: \"8f0d9b47-0865-402c-bfd5-a95e85236c27\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8"
Jul 13 20:24:49.806816 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.806674    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f0d9b47-0865-402c-bfd5-a95e85236c27-lib-modules\") pod \"perf-node-gather-daemonset-gtnw8\" (UID: \"8f0d9b47-0865-402c-bfd5-a95e85236c27\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8"
Jul 13 20:24:49.823469 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.823443    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf56k\" (UniqueName: \"kubernetes.io/projected/8f0d9b47-0865-402c-bfd5-a95e85236c27-kube-api-access-pf56k\") pod \"perf-node-gather-daemonset-gtnw8\" (UID: \"8f0d9b47-0865-402c-bfd5-a95e85236c27\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8"
Jul 13 20:24:49.931953 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:49.931915    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8"
Jul 13 20:24:50.123218 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:50.123030    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8"]
Jul 13 20:24:50.125695 ip-10-0-139-134 kubenswrapper[2569]: W0713 20:24:50.125666    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8f0d9b47_0865_402c_bfd5_a95e85236c27.slice/crio-efc79e9d5c4f7656c13df747298664ea3ca06e6e24730d7a5c04f8ead8ca11f2 WatchSource:0}: Error finding container efc79e9d5c4f7656c13df747298664ea3ca06e6e24730d7a5c04f8ead8ca11f2: Status 404 returned error can't find the container with id efc79e9d5c4f7656c13df747298664ea3ca06e6e24730d7a5c04f8ead8ca11f2
Jul 13 20:24:50.504433 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:50.504320    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8" event={"ID":"8f0d9b47-0865-402c-bfd5-a95e85236c27","Type":"ContainerStarted","Data":"d53e0c921e6d282c464e1d255b0bc30903fb9e0bdda61394ca56ab299ce58fe3"}
Jul 13 20:24:50.504433 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:50.504398    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8" event={"ID":"8f0d9b47-0865-402c-bfd5-a95e85236c27","Type":"ContainerStarted","Data":"efc79e9d5c4f7656c13df747298664ea3ca06e6e24730d7a5c04f8ead8ca11f2"}
Jul 13 20:24:50.504433 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:50.504430    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8"
Jul 13 20:24:50.527595 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:50.527537    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8" podStartSLOduration=1.527520131 podStartE2EDuration="1.527520131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:24:50.526496748 +0000 UTC m=+5712.040295956" watchObservedRunningTime="2026-07-13 20:24:50.527520131 +0000 UTC m=+5712.041319296"
Jul 13 20:24:51.665276 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:51.665229    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/1.log"
Jul 13 20:24:51.674495 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:51.674468    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-s92dx_a769d459-df42-47dc-8cfd-08a32d48bcea/console-operator/2.log"
Jul 13 20:24:52.255564 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:52.255527    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-9tnzg_92c2bd01-a76e-476c-b074-3a33fd725f90/download-server/0.log"
Jul 13 20:24:54.050561 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:54.050534    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xzzrg_c4789bc2-8df6-4746-b737-e6f4fd4a9f99/dns/0.log"
Jul 13 20:24:54.088061 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:54.088031    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xzzrg_c4789bc2-8df6-4746-b737-e6f4fd4a9f99/kube-rbac-proxy/0.log"
Jul 13 20:24:54.170970 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:54.170941    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vxf4w_3963e434-e4ab-4d73-b496-2472c7711891/dns-node-resolver/0.log"
Jul 13 20:24:54.808644 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:54.808611    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gdf7f_ddda9308-3395-4d7b-af21-a398125bd512/node-ca/0.log"
Jul 13 20:24:56.521608 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:56.521580    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-gtnw8"
Jul 13 20:24:56.603898 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:56.603871    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-586b57644b-q24lm_4fcfa1f0-d0ea-488a-a414-d09b8085aae5/router/0.log"
Jul 13 20:24:57.188084 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:57.188053    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9qzhl_502189f5-f517-4aa2-b359-07b8c5be3b9a/serve-healthcheck-canary/0.log"
Jul 13 20:24:57.724507 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:57.724472    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-tbqhd_8977c40b-f5c4-4aea-a441-6611214b30f7/insights-operator/0.log"
Jul 13 20:24:57.727102 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:57.727075    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-tbqhd_8977c40b-f5c4-4aea-a441-6611214b30f7/insights-operator/1.log"
Jul 13 20:24:57.787603 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:57.787569    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2h9jt_929eec72-b408-4af0-9f6f-8e140d648876/kube-rbac-proxy/0.log"
Jul 13 20:24:57.839047 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:57.839017    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2h9jt_929eec72-b408-4af0-9f6f-8e140d648876/exporter/0.log"
Jul 13 20:24:57.896499 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:24:57.896468    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2h9jt_929eec72-b408-4af0-9f6f-8e140d648876/extractor/0.log"
Jul 13 20:25:01.332523 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:25:01.332487    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5ff584b858-jl8gt_f345f616-e936-4b68-9497-0fa2dc3855c9/manager/0.log"
Jul 13 20:25:01.673175 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:25:01.673142    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-mrl5g_ebbf3625-7f4e-477c-9502-6bb3e5bb11d1/openshift-lws-operator/0.log"
Jul 13 20:25:01.699925 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:25:01.699890    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-586b57644b-q24lm_4fcfa1f0-d0ea-488a-a414-d09b8085aae5/router/0.log"
Jul 13 20:25:02.458217 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:25:02.458178    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-dc6dd9667-pq4h9_8b63f5d1-ddc1-4b49-9bd1-69133ae2919f/manager/0.log"
Jul 13 20:25:03.600671 ip-10-0-139-134 kubenswrapper[2569]: I0713 20:25:03.600642    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-586b57644b-q24lm_4fcfa1f0-d0ea-488a-a414-d09b8085aae5/router/0.log"