Jul 06 20:49:11.058975 ip-10-0-134-202 systemd[1]: Starting Kubernetes Kubelet...
Jul 06 20:49:11.432518 ip-10-0-134-202 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 20:49:11.432518 ip-10-0-134-202 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 06 20:49:11.432518 ip-10-0-134-202 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 20:49:11.432518 ip-10-0-134-202 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 06 20:49:11.433196 ip-10-0-134-202 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 20:49:11.433477 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.433319    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 06 20:49:11.437246 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437226    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:49:11.437246 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437245    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:49:11.437318 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437249    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:49:11.437318 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437253    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:49:11.437318 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437257    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:49:11.437318 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437260    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:49:11.437318 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437263    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:49:11.437318 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437266    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:49:11.437318 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437269    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:49:11.437318 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437280    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:49:11.437318 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437283    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:49:11.437318 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437286    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:49:11.437318 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437289    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:49:11.437318 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437292    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:49:11.437318 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437295    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:49:11.437318 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437298    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:49:11.437318 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437302    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:49:11.437318 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437304    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:49:11.437318 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437307    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:49:11.437318 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437310    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:49:11.437318 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437313    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:49:11.437318 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437315    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:49:11.437858 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437318    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:49:11.437858 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437321    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:49:11.437858 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437324    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:49:11.437858 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437328    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:49:11.437858 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437331    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:49:11.437858 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437334    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:49:11.437858 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437336    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:49:11.437858 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437339    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:49:11.437858 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437342    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:49:11.437858 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437344    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:49:11.437858 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437347    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:49:11.437858 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437350    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:49:11.437858 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437352    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:49:11.437858 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437356    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:49:11.437858 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437361    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:49:11.437858 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437365    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:49:11.437858 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437368    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:49:11.437858 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437370    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:49:11.437858 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437375    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:49:11.438349 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437378    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:49:11.438349 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437382    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:49:11.438349 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437384    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:49:11.438349 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437387    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:49:11.438349 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437390    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:49:11.438349 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437393    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:49:11.438349 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437395    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:49:11.438349 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437398    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:49:11.438349 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437401    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:49:11.438349 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437403    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:49:11.438349 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437406    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:49:11.438349 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437408    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:49:11.438349 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437411    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:49:11.438349 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437413    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:49:11.438349 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437417    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:49:11.438349 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437420    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:49:11.438349 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437424    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:49:11.438349 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437428    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:49:11.438349 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437431    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:49:11.438875 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437434    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:49:11.438875 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437437    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:49:11.438875 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437440    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:49:11.438875 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437443    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:49:11.438875 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437446    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:49:11.438875 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437449    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:49:11.438875 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437452    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:49:11.438875 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437454    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:49:11.438875 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437457    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:49:11.438875 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437460    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:49:11.438875 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437462    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:49:11.438875 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437466    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:49:11.438875 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437468    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:49:11.438875 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437471    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:49:11.438875 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437475    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:49:11.438875 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437478    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:49:11.438875 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437480    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:49:11.438875 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437483    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:49:11.438875 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437486    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:49:11.438875 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437489    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:49:11.439406 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437491    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:49:11.439406 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437494    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:49:11.439406 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437496    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:49:11.439406 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437499    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:49:11.439406 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437501    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:49:11.439406 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437504    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:49:11.439406 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437507    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:49:11.439406 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437511    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:49:11.439406 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437514    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:49:11.439406 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437517    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:49:11.439406 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437520    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:49:11.439406 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437523    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:49:11.439406 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437526    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:49:11.439406 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437529    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:49:11.439406 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437531    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:49:11.439406 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437534    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:49:11.439406 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437537    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:49:11.439406 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437540    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:49:11.439406 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.437542    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:49:11.439406 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438527    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:49:11.439932 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438537    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:49:11.439932 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438543    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:49:11.439932 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438546    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:49:11.439932 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438549    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:49:11.439932 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438553    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:49:11.439932 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438557    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:49:11.439932 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438560    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:49:11.439932 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438563    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:49:11.439932 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438566    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:49:11.439932 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438569    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:49:11.439932 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438572    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:49:11.439932 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438575    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:49:11.439932 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438578    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:49:11.439932 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438581    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:49:11.439932 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438584    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:49:11.439932 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438586    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:49:11.439932 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438589    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:49:11.439932 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438592    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:49:11.439932 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438595    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:49:11.440440 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438599    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:49:11.440440 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438602    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:49:11.440440 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438605    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:49:11.440440 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438609    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:49:11.440440 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438611    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:49:11.440440 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438614    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:49:11.440440 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438617    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:49:11.440440 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438619    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:49:11.440440 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438623    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:49:11.440440 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438626    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:49:11.440440 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438629    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:49:11.440440 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438632    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:49:11.440440 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438634    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:49:11.440440 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438637    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:49:11.440440 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438640    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:49:11.440440 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438643    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:49:11.440440 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438646    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:49:11.440440 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438649    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:49:11.440440 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438655    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:49:11.440440 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438659    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:49:11.440985 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438664    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:49:11.440985 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438668    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:49:11.440985 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438671    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:49:11.440985 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438674    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:49:11.440985 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438677    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:49:11.440985 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438680    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:49:11.440985 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438701    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:49:11.440985 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438704    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:49:11.440985 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438708    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:49:11.440985 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438711    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:49:11.440985 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438714    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:49:11.440985 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438717    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:49:11.440985 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438720    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:49:11.440985 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438723    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:49:11.440985 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438726    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:49:11.440985 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438729    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:49:11.440985 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438732    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:49:11.440985 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438734    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:49:11.440985 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438737    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:49:11.440985 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438740    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:49:11.441520 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438743    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:49:11.441520 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438745    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:49:11.441520 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438748    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:49:11.441520 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438751    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:49:11.441520 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438754    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:49:11.441520 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438756    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:49:11.441520 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438759    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:49:11.441520 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438762    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:49:11.441520 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438768    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:49:11.441520 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438772    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:49:11.441520 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438775    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:49:11.441520 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438779    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:49:11.441520 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438782    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:49:11.441520 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438786    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:49:11.441520 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438788    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:49:11.441520 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438791    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:49:11.441520 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438794    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:49:11.441520 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438797    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:49:11.441520 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438800    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:49:11.442043 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438803    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:49:11.442043 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438806    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:49:11.442043 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438809    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:49:11.442043 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438812    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:49:11.442043 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438814    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:49:11.442043 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438817    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:49:11.442043 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438820    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:49:11.442043 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438822    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:49:11.442043 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438825    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:49:11.442043 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438828    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:49:11.442043 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438831    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:49:11.442043 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438833    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:49:11.442043 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438836    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:49:11.442043 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438839    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:49:11.442043 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438842    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:49:11.442043 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438844    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:49:11.442043 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438847    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:49:11.442043 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438850    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:49:11.442043 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438852    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:49:11.442043 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.438855    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:49:11.442558 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439271    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:49:11.442558 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439277    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:49:11.442558 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439282    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:49:11.442558 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439286    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:49:11.442558 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439289    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:49:11.442558 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439292    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:49:11.442558 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439295    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:49:11.442558 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439298    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:49:11.442558 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439301    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:49:11.442558 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439303    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:49:11.442558 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439306    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:49:11.442558 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439311    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:49:11.442558 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439314    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:49:11.442558 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439317    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:49:11.442558 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439320    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:49:11.442558 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439323    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:49:11.442558 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439327    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:49:11.442558 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439331    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:49:11.442558 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439335    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:49:11.442558 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439338    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:49:11.443101 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439341    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:49:11.443101 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439344    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:49:11.443101 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439347    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:49:11.443101 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439350    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:49:11.443101 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439352    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:49:11.443101 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439355    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:49:11.443101 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439358    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:49:11.443101 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439360    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:49:11.443101 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439363    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:49:11.443101 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439366    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:49:11.443101 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439369    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:49:11.443101 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439372    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:49:11.443101 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439375    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:49:11.443101 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439377    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:49:11.443101 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439380    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:49:11.443101 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439383    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:49:11.443101 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439385    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:49:11.443101 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439392    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:49:11.443101 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439395    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:49:11.443624 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439397    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:49:11.443624 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439400    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:49:11.443624 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439403    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:49:11.443624 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439406    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:49:11.443624 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439410    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:49:11.443624 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439413    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:49:11.443624 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439416    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:49:11.443624 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439419    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:49:11.443624 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439422    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:49:11.443624 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439425    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:49:11.443624 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439428    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:49:11.443624 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439431    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:49:11.443624 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439433    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:49:11.443624 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439436    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:49:11.443624 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439439    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:49:11.443624 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439441    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:49:11.443624 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439444    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:49:11.443624 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439447    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:49:11.443624 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439450    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:49:11.443624 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439453    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:49:11.444168 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439456    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:49:11.444168 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439459    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:49:11.444168 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439461    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:49:11.444168 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439465    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:49:11.444168 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439471    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:49:11.444168 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439474    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:49:11.444168 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439477    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:49:11.444168 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439480    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:49:11.444168 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439484    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:49:11.444168 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439487    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:49:11.444168 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439489    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:49:11.444168 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439493    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:49:11.444168 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439496    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:49:11.444168 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439498    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:49:11.444168 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439501    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:49:11.444168 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439504    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:49:11.444168 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439508    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:49:11.444168 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439511    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:49:11.444168 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439514    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:49:11.444667 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439516    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:49:11.444667 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439519    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:49:11.444667 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439522    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:49:11.444667 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439525    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:49:11.444667 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439527    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:49:11.444667 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439530    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:49:11.444667 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439533    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:49:11.444667 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439536    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:49:11.444667 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439539    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:49:11.444667 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439541    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:49:11.444667 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439544    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:49:11.444667 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439547    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:49:11.444667 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439549    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:49:11.444667 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439552    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:49:11.444667 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439555    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:49:11.444667 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439557    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:49:11.444667 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439560    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:49:11.444667 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439564    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:49:11.444667 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439567    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:49:11.444667 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439569    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.439572    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439652    2559 flags.go:64] FLAG: --address="0.0.0.0"
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439671    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439680    2559 flags.go:64] FLAG: --anonymous-auth="true"
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439703    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439709    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439712    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439717    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439722    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439729    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439732    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439736    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439740    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439743    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439746    2559 flags.go:64] FLAG: --cgroup-root=""
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439750    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439753    2559 flags.go:64] FLAG: --client-ca-file=""
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439756    2559 flags.go:64] FLAG: --cloud-provider="external"
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439759    2559 flags.go:64] FLAG: --cluster-dns="[]"
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439764    2559 flags.go:64] FLAG: --cluster-domain=""
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439767    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439770    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439774    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 06 20:49:11.445245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439778    2559 flags.go:64] FLAG: --container-log-max-files="5"
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439782    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439786    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439789    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439793    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439796    2559 flags.go:64] FLAG: --contention-profiling="false"
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439800    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439804    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439807    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439810    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439814    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439817    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439821    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439824    2559 flags.go:64] FLAG: --enable-load-reader="false"
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439827    2559 flags.go:64] FLAG: --enable-server="true"
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439831    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439835    2559 flags.go:64] FLAG: --event-burst="100"
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439838    2559 flags.go:64] FLAG: --event-qps="50"
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439843    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439846    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439849    2559 flags.go:64] FLAG: --eviction-hard=""
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439854    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439857    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439860    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439863    2559 flags.go:64] FLAG: --eviction-soft=""
Jul 06 20:49:11.445874 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439866    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439870    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439873    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439877    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439880    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439883    2559 flags.go:64] FLAG: --fail-swap-on="true"
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439886    2559 flags.go:64] FLAG: --feature-gates=""
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439890    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439893    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439897    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439900    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439903    2559 flags.go:64] FLAG: --healthz-port="10248"
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439906    2559 flags.go:64] FLAG: --help="false"
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439912    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-134-202.ec2.internal"
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439915    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439918    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439922    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439925    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439929    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439932    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439936    2559 flags.go:64] FLAG: --image-service-endpoint=""
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439939    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439942    2559 flags.go:64] FLAG: --kube-api-burst="100"
Jul 06 20:49:11.446551 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439945    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439949    2559 flags.go:64] FLAG: --kube-api-qps="50"
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439953    2559 flags.go:64] FLAG: --kube-reserved=""
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439956    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439959    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439962    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439965    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439968    2559 flags.go:64] FLAG: --lock-file=""
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439971    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439974    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439977    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439983    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439986    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439989    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439992    2559 flags.go:64] FLAG: --logging-format="text"
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439994    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.439998    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440001    2559 flags.go:64] FLAG: --manifest-url=""
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440004    2559 flags.go:64] FLAG: --manifest-url-header=""
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440008    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440012    2559 flags.go:64] FLAG: --max-open-files="1000000"
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440015    2559 flags.go:64] FLAG: --max-pods="110"
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440020    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440024    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440027    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 06 20:49:11.447185 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440030    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440033    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440036    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440039    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440047    2559 flags.go:64] FLAG: --node-status-max-images="50"
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440050    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440053    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440056    2559 flags.go:64] FLAG: --pod-cidr=""
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440060    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440067    2559 flags.go:64] FLAG: --pod-manifest-path=""
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440070    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440074    2559 flags.go:64] FLAG: --pods-per-core="0"
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440077    2559 flags.go:64] FLAG: --port="10250"
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440080    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440083    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c665fd73f4c94ad2"
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440086    2559 flags.go:64] FLAG: --qos-reserved=""
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440090    2559 flags.go:64] FLAG: --read-only-port="10255"
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440093    2559 flags.go:64] FLAG: --register-node="true"
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440096    2559 flags.go:64] FLAG: --register-with-taints=""
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440100    2559 flags.go:64] FLAG: --registry-burst="10"
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440103    2559 flags.go:64] FLAG: --registry-qps="5"
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440106    2559 flags.go:64] FLAG: --reserved-cpus=""
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440109    2559 flags.go:64] FLAG: --reserved-memory=""
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440113    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440116    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 06 20:49:11.447857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440119    2559 flags.go:64] FLAG: --rotate-certificates="false"
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440122    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440125    2559 flags.go:64] FLAG: --runonce="false"
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440128    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440134    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440137    2559 flags.go:64] FLAG: --seccomp-default="false"
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440140    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440143    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440147    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440150    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440154    2559 flags.go:64] FLAG: --storage-driver-password="root"
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440157    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440160    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440163    2559 flags.go:64] FLAG: --storage-driver-user="root"
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440166    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440170    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440173    2559 flags.go:64] FLAG: --system-cgroups=""
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440177    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440183    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440186    2559 flags.go:64] FLAG: --tls-cert-file=""
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440189    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440194    2559 flags.go:64] FLAG: --tls-min-version=""
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440197    2559 flags.go:64] FLAG: --tls-private-key-file=""
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440200    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440203    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 06 20:49:11.448517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440207    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 06 20:49:11.449268 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440212    2559 flags.go:64] FLAG: --v="2"
Jul 06 20:49:11.449268 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440218    2559 flags.go:64] FLAG: --version="false"
Jul 06 20:49:11.449268 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440225    2559 flags.go:64] FLAG: --vmodule=""
Jul 06 20:49:11.449268 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440270    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 06 20:49:11.449268 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.440279    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 06 20:49:11.449268 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440420    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:49:11.449268 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440425    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:49:11.449268 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440428    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:49:11.449268 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440432    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:49:11.449268 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440435    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:49:11.449268 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440438    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:49:11.449268 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440441    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:49:11.449268 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440444    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:49:11.449268 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440447    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:49:11.449268 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440451    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:49:11.449268 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440454    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:49:11.449268 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440458    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:49:11.449268 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440462    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:49:11.449268 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440466    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:49:11.449268 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440469    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:49:11.449268 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440473    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:49:11.449846 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440477    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:49:11.449846 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440481    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:49:11.449846 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440484    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:49:11.449846 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440487    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:49:11.449846 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440490    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:49:11.449846 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440494    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:49:11.449846 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440498    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:49:11.449846 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440502    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:49:11.449846 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440505    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:49:11.449846 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440508    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:49:11.449846 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440510    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:49:11.449846 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440513    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:49:11.449846 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440515    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:49:11.449846 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440518    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:49:11.449846 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440522    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:49:11.449846 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440526    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:49:11.449846 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440529    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:49:11.449846 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440532    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:49:11.449846 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440534    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:49:11.450388 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440537    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:49:11.450388 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440540    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:49:11.450388 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440543    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:49:11.450388 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440546    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:49:11.450388 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440548    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:49:11.450388 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440551    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:49:11.450388 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440554    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:49:11.450388 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440557    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:49:11.450388 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440560    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:49:11.450388 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440563    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:49:11.450388 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440566    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:49:11.450388 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440569    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:49:11.450388 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440571    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:49:11.450388 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440574    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:49:11.450388 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440577    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:49:11.450388 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440580    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:49:11.450388 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440582    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:49:11.450388 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440586    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:49:11.450388 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440588    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:49:11.450900 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440591    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:49:11.450900 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440594    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:49:11.450900 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440596    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:49:11.450900 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440599    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:49:11.450900 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440602    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:49:11.450900 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440605    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:49:11.450900 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440607    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:49:11.450900 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440610    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:49:11.450900 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440614    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:49:11.450900 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440617    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:49:11.450900 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440620    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:49:11.450900 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440623    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:49:11.450900 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440625    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:49:11.450900 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440628    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:49:11.450900 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440631    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:49:11.450900 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440633    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:49:11.450900 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440636    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:49:11.450900 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440651    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:49:11.450900 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440654    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:49:11.450900 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440657    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:49:11.451430 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440660    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:49:11.451430 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440663    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:49:11.451430 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440666    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:49:11.451430 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440669    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:49:11.451430 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440672    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:49:11.451430 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440675    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:49:11.451430 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440678    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:49:11.451430 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440681    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:49:11.451430 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440684    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:49:11.451430 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440687    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:49:11.451430 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440689    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:49:11.451430 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440692    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:49:11.451430 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440695    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:49:11.451430 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440698    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:49:11.451430 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440701    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:49:11.451430 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440703    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:49:11.451430 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440706    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:49:11.451430 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440708    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:49:11.451430 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440711    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:49:11.451430 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440714    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:49:11.451951 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440720    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:49:11.451951 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440723    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:49:11.451951 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440727    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:49:11.451951 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440730    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:49:11.451951 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.440733    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:49:11.451951 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.441336    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 20:49:11.451951 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.450330    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 06 20:49:11.451951 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.450360    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 06 20:49:11.451951 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.450378    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 06 20:49:11.451951 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.450385    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 06 20:49:11.451951 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450456    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:49:11.451951 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450462    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:49:11.451951 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450466    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:49:11.451951 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450469    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:49:11.451951 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450472    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:49:11.452378 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450476    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:49:11.452378 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450479    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:49:11.452378 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450482    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:49:11.452378 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450485    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:49:11.452378 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450489    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:49:11.452378 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450492    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:49:11.452378 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450495    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:49:11.452378 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450497    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:49:11.452378 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450500    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:49:11.452378 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450503    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:49:11.452378 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450505    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:49:11.452378 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450508    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:49:11.452378 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450511    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:49:11.452378 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450514    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:49:11.452378 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450517    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:49:11.452378 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450520    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:49:11.452378 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450522    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:49:11.452378 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450525    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:49:11.452378 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450529    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:49:11.452887 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450534    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:49:11.452887 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450538    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:49:11.452887 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450540    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:49:11.452887 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450545    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:49:11.452887 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450548    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:49:11.452887 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450552    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:49:11.452887 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450556    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:49:11.452887 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450559    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:49:11.452887 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450562    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:49:11.452887 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450565    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:49:11.452887 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450568    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:49:11.452887 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450571    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:49:11.452887 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450574    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:49:11.452887 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450578    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:49:11.452887 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450581    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:49:11.452887 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450584    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:49:11.452887 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450586    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:49:11.452887 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450589    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:49:11.452887 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450592    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:49:11.453382 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450595    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:49:11.453382 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450598    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:49:11.453382 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450601    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:49:11.453382 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450604    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:49:11.453382 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450607    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:49:11.453382 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450610    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:49:11.453382 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450613    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:49:11.453382 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450615    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:49:11.453382 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450618    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:49:11.453382 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450621    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:49:11.453382 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450624    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:49:11.453382 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450627    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:49:11.453382 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450652    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:49:11.453382 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450657    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:49:11.453382 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450660    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:49:11.453382 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450663    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:49:11.453382 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450666    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:49:11.453382 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450669    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:49:11.453382 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450673    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:49:11.453382 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450676    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:49:11.453944 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450679    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:49:11.453944 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450682    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:49:11.453944 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450685    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:49:11.453944 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450687    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:49:11.453944 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450690    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:49:11.453944 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450693    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:49:11.453944 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450696    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:49:11.453944 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450698    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:49:11.453944 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450701    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:49:11.453944 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450704    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:49:11.453944 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450707    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:49:11.453944 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450709    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:49:11.453944 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450712    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:49:11.453944 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450715    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:49:11.453944 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450717    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:49:11.453944 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450720    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:49:11.453944 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450723    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:49:11.453944 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450726    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:49:11.453944 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450729    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:49:11.453944 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450731    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:49:11.454538 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450734    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:49:11.454538 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450737    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:49:11.454538 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450740    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:49:11.454538 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450742    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:49:11.454538 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450747    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:49:11.454538 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450750    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:49:11.454538 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450753    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:49:11.454538 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450757    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:49:11.454538 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450760    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:49:11.454538 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450764    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:49:11.454538 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450767    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:49:11.454538 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450770    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:49:11.454538 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450773    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:49:11.454538 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450775    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:49:11.454538 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450778    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:49:11.454538 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450781    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:49:11.454998 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.450787    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 20:49:11.454998 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450912    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:49:11.454998 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450917    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:49:11.454998 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450921    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:49:11.454998 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450924    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:49:11.454998 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450927    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:49:11.454998 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450930    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:49:11.454998 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450934    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:49:11.454998 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450937    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:49:11.454998 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450940    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:49:11.454998 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450943    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:49:11.454998 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450946    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:49:11.454998 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450949    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:49:11.454998 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450952    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:49:11.454998 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450955    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:49:11.455409 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450958    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:49:11.455409 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450961    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:49:11.455409 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450964    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:49:11.455409 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450967    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:49:11.455409 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450970    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:49:11.455409 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450972    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:49:11.455409 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450975    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:49:11.455409 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450978    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:49:11.455409 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450981    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:49:11.455409 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450983    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:49:11.455409 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450988    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:49:11.455409 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450991    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:49:11.455409 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450994    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:49:11.455409 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.450997    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:49:11.455409 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451000    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:49:11.455409 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451002    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:49:11.455409 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451005    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:49:11.455409 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451008    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:49:11.455409 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451010    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:49:11.455409 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451013    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:49:11.456030 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451016    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:49:11.456030 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451019    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:49:11.456030 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451021    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:49:11.456030 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451024    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:49:11.456030 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451027    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:49:11.456030 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451030    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:49:11.456030 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451032    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:49:11.456030 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451035    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:49:11.456030 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451038    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:49:11.456030 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451040    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:49:11.456030 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451043    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:49:11.456030 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451046    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:49:11.456030 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451048    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:49:11.456030 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451051    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:49:11.456030 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451054    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:49:11.456030 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451057    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:49:11.456030 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451060    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:49:11.456030 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451063    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:49:11.456030 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451066    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:49:11.456543 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451070    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:49:11.456543 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451073    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:49:11.456543 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451076    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:49:11.456543 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451080    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:49:11.456543 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451083    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:49:11.456543 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451086    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:49:11.456543 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451088    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:49:11.456543 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451091    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:49:11.456543 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451094    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:49:11.456543 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451097    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:49:11.456543 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451100    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:49:11.456543 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451102    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:49:11.456543 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451105    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:49:11.456543 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451108    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:49:11.456543 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451110    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:49:11.456543 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451113    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:49:11.456543 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451116    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:49:11.456543 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451119    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:49:11.456543 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451121    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:49:11.457092 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451124    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:49:11.457092 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451127    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:49:11.457092 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451129    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:49:11.457092 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451132    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:49:11.457092 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451135    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:49:11.457092 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451138    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:49:11.457092 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451140    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:49:11.457092 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451143    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:49:11.457092 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451147    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:49:11.457092 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451151    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:49:11.457092 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451154    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:49:11.457092 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451157    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:49:11.457092 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451160    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:49:11.457092 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451163    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:49:11.457092 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451166    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:49:11.457092 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451169    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:49:11.457092 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451171    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:49:11.457092 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451174    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:49:11.457092 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451178    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:49:11.457612 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451180    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:49:11.457612 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451183    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:49:11.457612 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451187    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:49:11.457612 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451190    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:49:11.457612 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451193    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:49:11.457612 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451197    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:49:11.457612 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451200    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:49:11.457612 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:11.451203    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:49:11.457612 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.451208    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 20:49:11.457612 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.451339    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 06 20:49:11.457612 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.453517    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 06 20:49:11.457612 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.454889    2559 server.go:1019] "Starting client certificate rotation"
Jul 06 20:49:11.457612 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.454994    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 20:49:11.458003 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.455029    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 20:49:11.458003 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.457550    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 20:49:11.463115 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.463090    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 20:49:11.476016 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.475998    2559 log.go:25] "Validated CRI v1 runtime API"
Jul 06 20:49:11.478061 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.478046    2559 log.go:25] "Validated CRI v1 image API"
Jul 06 20:49:11.478919 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.478898    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 06 20:49:11.481645 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.481622    2559 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7fbb3cd6-76dd-4955-96aa-cb8607b8c1c3:/dev/nvme0n1p3 c7615ab5-bc4a-4f48-b166-a3545de17f4c:/dev/nvme0n1p4]
Jul 06 20:49:11.481735 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.481645    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 06 20:49:11.487212 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.487099    2559 manager.go:216] Machine: {Timestamp:2026-07-06 20:49:11.485516106 +0000 UTC m=+0.320403488 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098178 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec263c3b448051bb0ebe91cae78b6894 SystemUUID:ec263c3b-4480-51bb-0ebe-91cae78b6894 BootID:d30a0ea1-c56e-4573-8d08-9ed9e3a9ce96 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b6:d7:65:80:19 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b6:d7:65:80:19 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:75:6e:66:ab:5f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 06 20:49:11.487212 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.487209    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 06 20:49:11.487326 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.487289    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 06 20:49:11.489823 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.489793    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 06 20:49:11.489963 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.489826    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-202.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 06 20:49:11.490619 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.490609    2559 topology_manager.go:138] "Creating topology manager with none policy"
Jul 06 20:49:11.490650 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.490622    2559 container_manager_linux.go:309] "Creating device plugin manager"
Jul 06 20:49:11.490650 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.490630    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 20:49:11.491313 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.491301    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 20:49:11.492568 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.492558    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 06 20:49:11.495789 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.495776    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 20:49:11.495910 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.495901    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 06 20:49:11.496824 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.496814    2559 kubelet.go:486] "Attempting to sync node with API server"
Jul 06 20:49:11.496864 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.496831    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 06 20:49:11.496864 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.496847    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 06 20:49:11.496864 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.496859    2559 kubelet.go:398] "Adding apiserver pod source"
Jul 06 20:49:11.496967 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.496881    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 06 20:49:11.497275 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.497244    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 20:49:11.497275 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.497266    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 20:49:11.499487 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.499468    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 06 20:49:11.500264 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.500246    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 20:49:11.501970 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.501945    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 06 20:49:11.502069 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.501999    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 06 20:49:11.503754 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.503739    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 06 20:49:11.503819 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.503775    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 06 20:49:11.503819 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.503784    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 06 20:49:11.503819 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.503792    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 06 20:49:11.503819 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.503802    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 06 20:49:11.503819 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.503809    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 06 20:49:11.503819 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.503819    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 06 20:49:11.504008 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.503826    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 06 20:49:11.504008 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.503833    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 06 20:49:11.504008 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.503844    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 06 20:49:11.504008 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.503859    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 06 20:49:11.504008 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.503866    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 06 20:49:11.504862 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.504621    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 06 20:49:11.504904 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.504875    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 06 20:49:11.508461 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.508446    2559 server.go:1268] "Started kubelet"
Jul 06 20:49:11.508571 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.508540    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 06 20:49:11.508711 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.508651    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 06 20:49:11.508769 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.508735    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 06 20:49:11.509405 ip-10-0-134-202 systemd[1]: Started Kubernetes Kubelet.
Jul 06 20:49:11.512160 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.510947    2559 server.go:310] "Adding debug handlers to kubelet server"
Jul 06 20:49:11.512160 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.512039    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-202.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 06 20:49:11.512160 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:11.512125    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-202.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 06 20:49:11.512455 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:11.512426    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 06 20:49:11.514962 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.514945    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 06 20:49:11.515419 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:11.515395    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 06 20:49:11.519964 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:11.519082    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-202.ec2.internal.18bfce6e8368e034  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-202.ec2.internal,UID:ip-10-0-134-202.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-202.ec2.internal,},FirstTimestamp:2026-07-06 20:49:11.508418612 +0000 UTC m=+0.343305971,LastTimestamp:2026-07-06 20:49:11.508418612 +0000 UTC m=+0.343305971,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-202.ec2.internal,}"
Jul 06 20:49:11.523787 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.523764    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 06 20:49:11.523787 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.523780    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 06 20:49:11.523971 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:11.523844    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found"
Jul 06 20:49:11.524362 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.524341    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 06 20:49:11.524362 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.524364    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 06 20:49:11.524507 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.524458    2559 reconstruct.go:97] "Volume reconstruction finished"
Jul 06 20:49:11.524507 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.524469    2559 reconciler.go:29] "Reconciler: start to sync state"
Jul 06 20:49:11.524507 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.524484    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 06 20:49:11.524843 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.524813    2559 factory.go:55] Registering systemd factory
Jul 06 20:49:11.524843 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.524831    2559 factory.go:223] Registration of the systemd container factory successfully
Jul 06 20:49:11.525093 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.525074    2559 factory.go:153] Registering CRI-O factory
Jul 06 20:49:11.525093 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.525095    2559 factory.go:223] Registration of the crio container factory successfully
Jul 06 20:49:11.525270 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.525164    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 06 20:49:11.525270 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.525188    2559 factory.go:103] Registering Raw factory
Jul 06 20:49:11.525270 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.525203    2559 manager.go:1196] Started watching for new ooms in manager
Jul 06 20:49:11.525922 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.525907    2559 manager.go:349] Starting recovery of all containers
Jul 06 20:49:11.527145 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:11.527119    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-202.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 06 20:49:11.528231 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:11.528204    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 06 20:49:11.539726 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.539592    2559 manager.go:354] Recovery completed
Jul 06 20:49:11.544423 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.544409    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:49:11.546422 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.546396    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:49:11.546520 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.546440    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:49:11.546520 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.546455    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:49:11.547012 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.546995    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 06 20:49:11.547012 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.547010    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 06 20:49:11.547127 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.547028    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 20:49:11.547127 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.547010    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:49:11.547882 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.547866    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:49:11.547957 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.547897    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:49:11.547957 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.547911    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:49:11.548366 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.548353    2559 policy_none.go:49] "None policy: Start"
Jul 06 20:49:11.548418 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.548370    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 06 20:49:11.548418 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.548403    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 06 20:49:11.549438 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.549427    2559 policy_none.go:47] "Start"
Jul 06 20:49:11.560082 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:11.560015    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-202.ec2.internal.18bfce6e85acc342  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-202.ec2.internal,UID:ip-10-0-134-202.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-202.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-202.ec2.internal,},FirstTimestamp:2026-07-06 20:49:11.546422082 +0000 UTC m=+0.381309445,LastTimestamp:2026-07-06 20:49:11.546422082 +0000 UTC m=+0.381309445,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-202.ec2.internal,}"
Jul 06 20:49:11.563805 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.563787    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ddffv"
Jul 06 20:49:11.570362 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:11.570295    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-202.ec2.internal.18bfce6e85ad23ce  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-202.ec2.internal,UID:ip-10-0-134-202.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-202.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-202.ec2.internal,},FirstTimestamp:2026-07-06 20:49:11.546446798 +0000 UTC m=+0.381334158,LastTimestamp:2026-07-06 20:49:11.546446798 +0000 UTC m=+0.381334158,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-202.ec2.internal,}"
Jul 06 20:49:11.585658 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:11.585596    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-202.ec2.internal.18bfce6e85ad5ab1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-202.ec2.internal,UID:ip-10-0-134-202.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-202.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-202.ec2.internal,},FirstTimestamp:2026-07-06 20:49:11.546460849 +0000 UTC m=+0.381348211,LastTimestamp:2026-07-06 20:49:11.546460849 +0000 UTC m=+0.381348211,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-202.ec2.internal,}"
Jul 06 20:49:11.585992 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.585976    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ddffv"
Jul 06 20:49:11.593259 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.593247    2559 manager.go:337] "Starting Device Plugin manager"
Jul 06 20:49:11.593340 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:11.593282    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 06 20:49:11.593340 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.593292    2559 server.go:87] "Starting device plugin registration server"
Jul 06 20:49:11.593557 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.593539    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 06 20:49:11.593680 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.593556    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 06 20:49:11.593794 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.593775    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 06 20:49:11.593895 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.593871    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 06 20:49:11.593895 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.593884    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 06 20:49:11.594481 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:11.594453    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 06 20:49:11.594554 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:11.594521    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-202.ec2.internal\" not found"
Jul 06 20:49:11.649980 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.649908    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 06 20:49:11.651186 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.651166    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 06 20:49:11.651244 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.651217    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 06 20:49:11.651244 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.651242    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 06 20:49:11.651332 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:11.651292    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 06 20:49:11.653738 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.653717    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:49:11.694851 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.694816    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:49:11.695744 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.695729    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:49:11.695809 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.695762    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:49:11.695809 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.695773    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:49:11.695809 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.695796    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-202.ec2.internal"
Jul 06 20:49:11.705253 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.705238    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-202.ec2.internal"
Jul 06 20:49:11.705302 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:11.705262    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-202.ec2.internal\": node \"ip-10-0-134-202.ec2.internal\" not found"
Jul 06 20:49:11.733118 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:11.733094    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found"
Jul 06 20:49:11.751969 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.751946    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal"]
Jul 06 20:49:11.752079 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.752016    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:49:11.752938 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.752922    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:49:11.753012 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.752955    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:49:11.753012 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.752967    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:49:11.754533 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.754520    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:49:11.754666 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.754653    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal"
Jul 06 20:49:11.754724 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.754685    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:49:11.755265 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.755244    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:49:11.755265 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.755256    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:49:11.755402 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.755276    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:49:11.755402 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.755282    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:49:11.755402 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.755289    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:49:11.755402 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.755292    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:49:11.756730 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.756714    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal"
Jul 06 20:49:11.756824 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.756743    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:49:11.757527 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.757510    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:49:11.757602 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.757537    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:49:11.757602 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.757547    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:49:11.776306 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:11.776288    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-202.ec2.internal\" not found" node="ip-10-0-134-202.ec2.internal"
Jul 06 20:49:11.783705 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:11.783670    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-202.ec2.internal\" not found" node="ip-10-0-134-202.ec2.internal"
Jul 06 20:49:11.826706 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.826680    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3ee03c06c928493e4823bc059365a1f8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal\" (UID: \"3ee03c06c928493e4823bc059365a1f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal"
Jul 06 20:49:11.826806 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.826711    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ee03c06c928493e4823bc059365a1f8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal\" (UID: \"3ee03c06c928493e4823bc059365a1f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal"
Jul 06 20:49:11.826806 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.826729    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b12aea9bf8b99530f8679e8c6d1d16c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-202.ec2.internal\" (UID: \"4b12aea9bf8b99530f8679e8c6d1d16c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal"
Jul 06 20:49:11.833224 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:11.833205    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found"
Jul 06 20:49:11.927063 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.926993    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3ee03c06c928493e4823bc059365a1f8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal\" (UID: \"3ee03c06c928493e4823bc059365a1f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal"
Jul 06 20:49:11.927063 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.927019    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3ee03c06c928493e4823bc059365a1f8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal\" (UID: \"3ee03c06c928493e4823bc059365a1f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal"
Jul 06 20:49:11.927182 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.927076    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ee03c06c928493e4823bc059365a1f8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal\" (UID: \"3ee03c06c928493e4823bc059365a1f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal"
Jul 06 20:49:11.927182 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.927097    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b12aea9bf8b99530f8679e8c6d1d16c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-202.ec2.internal\" (UID: \"4b12aea9bf8b99530f8679e8c6d1d16c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal"
Jul 06 20:49:11.927182 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.927141    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b12aea9bf8b99530f8679e8c6d1d16c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-202.ec2.internal\" (UID: \"4b12aea9bf8b99530f8679e8c6d1d16c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal"
Jul 06 20:49:11.927182 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:11.927166    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ee03c06c928493e4823bc059365a1f8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal\" (UID: \"3ee03c06c928493e4823bc059365a1f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal"
Jul 06 20:49:11.933483 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:11.933459    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found"
Jul 06 20:49:12.033750 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:12.033717    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found"
Jul 06 20:49:12.082581 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:12.082557    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal"
Jul 06 20:49:12.087602 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:12.087581    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal"
Jul 06 20:49:12.133916 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:12.133889    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found"
Jul 06 20:49:12.234929 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:12.234862    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found"
Jul 06 20:49:12.335227 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:12.335196    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found"
Jul 06 20:49:12.435627 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:12.435583    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found"
Jul 06 20:49:12.454442 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:12.454412    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 06 20:49:12.454641 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:12.454599    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 20:49:12.536505 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:12.536469    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found"
Jul 06 20:49:12.569007 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:12.568980    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:49:12.570416 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:12.570398    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:49:12.570539 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:12.570435    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:49:12.570539 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:12.570450    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:49:12.587868 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:12.587829    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 20:44:12 +0000 UTC" deadline="2028-02-12 05:30:25.478337984 +0000 UTC"
Jul 06 20:49:12.587868 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:12.587858    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14048h41m12.890483234s"
Jul 06 20:49:12.613718 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:12.613661    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b12aea9bf8b99530f8679e8c6d1d16c.slice/crio-50b675dfab0c81e792484538e25a720dda03dc5dd1bfb9794dd8441b5c996402 WatchSource:0}: Error finding container 50b675dfab0c81e792484538e25a720dda03dc5dd1bfb9794dd8441b5c996402: Status 404 returned error can't find the container with id 50b675dfab0c81e792484538e25a720dda03dc5dd1bfb9794dd8441b5c996402
Jul 06 20:49:12.614178 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:12.614160    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee03c06c928493e4823bc059365a1f8.slice/crio-db27aa555a50c3554b0ce41f78a7bb27805b7bc86c15c5cfcaa362b17fb6190b WatchSource:0}: Error finding container db27aa555a50c3554b0ce41f78a7bb27805b7bc86c15c5cfcaa362b17fb6190b: Status 404 returned error can't find the container with id db27aa555a50c3554b0ce41f78a7bb27805b7bc86c15c5cfcaa362b17fb6190b
Jul 06 20:49:12.620321 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:12.620304    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 20:49:12.627668 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:12.627649    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:49:12.636815 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:12.636796    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found"
Jul 06 20:49:12.645818 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:12.645800    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:49:12.654030 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:12.653991    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal" event={"ID":"4b12aea9bf8b99530f8679e8c6d1d16c","Type":"ContainerStarted","Data":"50b675dfab0c81e792484538e25a720dda03dc5dd1bfb9794dd8441b5c996402"}
Jul 06 20:49:12.655071 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:12.655051    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" event={"ID":"3ee03c06c928493e4823bc059365a1f8","Type":"ContainerStarted","Data":"db27aa555a50c3554b0ce41f78a7bb27805b7bc86c15c5cfcaa362b17fb6190b"}
Jul 06 20:49:12.724875 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:12.724850    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal"
Jul 06 20:49:12.737083 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:12.737061    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 20:49:12.738055 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:12.738039    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal"
Jul 06 20:49:12.755574 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:12.755552    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 20:49:13.044024 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.043998    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:49:13.498847 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.498769    2559 apiserver.go:52] "Watching apiserver"
Jul 06 20:49:13.509149 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.509124    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 06 20:49:13.510479 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.510456    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-khnhx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal","openshift-multus/network-metrics-daemon-ckj9h","openshift-network-operator/iptables-alerter-5bp9d","openshift-ovn-kubernetes/ovnkube-node-pt6t9","kube-system/konnectivity-agent-2lhx7","kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal","openshift-cluster-node-tuning-operator/tuned-nmhcb","openshift-dns/node-resolver-mzmqn","openshift-image-registry/node-ca-rbr6h","openshift-multus/multus-46dgr","openshift-multus/multus-additional-cni-plugins-wf6lk","openshift-network-diagnostics/network-check-target-p9bcs"]
Jul 06 20:49:13.512019 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.511995    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2lhx7"
Jul 06 20:49:13.513160 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.513134    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.514864 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.514440    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 06 20:49:13.514864 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.514729    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 06 20:49:13.515023 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.514965    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5cvm7\""
Jul 06 20:49:13.515612 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.515593    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:13.515723 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:13.515678    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9bcs" podUID="806f1f1d-5cea-4e14-b4e8-a293bd4b3393"
Jul 06 20:49:13.517288 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.516838    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pnmdp\""
Jul 06 20:49:13.517288 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.517066    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 06 20:49:13.517288 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.517254    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 06 20:49:13.517603 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.517544    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 06 20:49:13.517870 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.517854    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 06 20:49:13.521468 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.521174    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 06 20:49:13.521577 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.521495    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:13.521650 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:13.521585    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckj9h" podUID="ad24b800-9783-47dc-aaa1-4d2bbc01afb8"
Jul 06 20:49:13.522993 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.521701    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5bp9d"
Jul 06 20:49:13.522993 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.521776    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.522993 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.522926    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.524885 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.524860    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 06 20:49:13.525952 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.525924    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 06 20:49:13.525952 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.525940    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rbr6h"
Jul 06 20:49:13.527023 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.526300    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vmbml\""
Jul 06 20:49:13.527023 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.526410    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hq98n\""
Jul 06 20:49:13.527023 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.526777    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:49:13.527177 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.526998    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 06 20:49:13.527177 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.527103    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 06 20:49:13.527389 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.527368    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 06 20:49:13.527540 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.527444    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 06 20:49:13.527742 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.527725    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xlrgk\""
Jul 06 20:49:13.527836 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.527801    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 06 20:49:13.528984 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.527974    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 06 20:49:13.528984 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.528106    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 06 20:49:13.528984 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.528193    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 06 20:49:13.528984 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.528580    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.532399 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.532380    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-phll7\""
Jul 06 20:49:13.532690 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.532671    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 06 20:49:13.532867 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.532851    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 06 20:49:13.533104 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.533085    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pqdzp\""
Jul 06 20:49:13.533213 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.533140    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 06 20:49:13.534521 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.534503    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 06 20:49:13.535169 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.535149    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 06 20:49:13.535389 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.535370    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 06 20:49:13.535624 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.535600    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-node-log\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.535733 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.535639    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.535733 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.535688    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ffd4833a-2f6e-4714-872b-bde81197a610-multus-daemon-config\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.535733 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.535716    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkzpl\" (UniqueName: \"kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl\") pod \"network-check-target-p9bcs\" (UID: \"806f1f1d-5cea-4e14-b4e8-a293bd4b3393\") " pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:13.535900 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.535741    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-host-run-netns\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.535900 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.535766    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5md4t\" (UniqueName: \"kubernetes.io/projected/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-kube-api-access-5md4t\") pod \"network-metrics-daemon-ckj9h\" (UID: \"ad24b800-9783-47dc-aaa1-4d2bbc01afb8\") " pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:13.535900 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.535790    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-host-run-k8s-cni-cncf-io\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.535900 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.535812    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-multus-conf-dir\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.535900 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.535848    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-host-run-multus-certs\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.536153 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.535873    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/575bca76-00f8-4a4b-af45-354830d0aa45-socket-dir\") pod \"aws-ebs-csi-driver-node-gscqt\" (UID: \"575bca76-00f8-4a4b-af45-354830d0aa45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.536153 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.535926    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3ad00703-a997-4d17-b251-09093cef791f-iptables-alerter-script\") pod \"iptables-alerter-5bp9d\" (UID: \"3ad00703-a997-4d17-b251-09093cef791f\") " pod="openshift-network-operator/iptables-alerter-5bp9d"
Jul 06 20:49:13.536153 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.535955    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.536153 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.535981    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-cnibin\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.536153 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536009    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-host-cni-netd\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.536153 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536033    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-ovnkube-script-lib\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.536153 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536061    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-systemd-units\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.536153 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536086    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-run-ovn\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.536153 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536110    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-os-release\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.536153 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536134    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zrw2\" (UniqueName: \"kubernetes.io/projected/ffd4833a-2f6e-4714-872b-bde81197a610-kube-api-access-2zrw2\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.536734 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536170    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d4b9056-4c7c-43fd-a2b6-6d617f5cf503-host\") pod \"node-ca-rbr6h\" (UID: \"5d4b9056-4c7c-43fd-a2b6-6d617f5cf503\") " pod="openshift-image-registry/node-ca-rbr6h"
Jul 06 20:49:13.536734 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536198    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/575bca76-00f8-4a4b-af45-354830d0aa45-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gscqt\" (UID: \"575bca76-00f8-4a4b-af45-354830d0aa45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.536734 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536245    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85jgm\" (UniqueName: \"kubernetes.io/projected/575bca76-00f8-4a4b-af45-354830d0aa45-kube-api-access-85jgm\") pod \"aws-ebs-csi-driver-node-gscqt\" (UID: \"575bca76-00f8-4a4b-af45-354830d0aa45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.536734 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536273    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8w9w\" (UniqueName: \"kubernetes.io/projected/3ad00703-a997-4d17-b251-09093cef791f-kube-api-access-j8w9w\") pod \"iptables-alerter-5bp9d\" (UID: \"3ad00703-a997-4d17-b251-09093cef791f\") " pod="openshift-network-operator/iptables-alerter-5bp9d"
Jul 06 20:49:13.536734 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536301    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk5xx\" (UniqueName: \"kubernetes.io/projected/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-kube-api-access-jk5xx\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.536734 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536334    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-host-run-netns\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.536734 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536365    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-host-var-lib-cni-bin\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.536734 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536396    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/575bca76-00f8-4a4b-af45-354830d0aa45-sys-fs\") pod \"aws-ebs-csi-driver-node-gscqt\" (UID: \"575bca76-00f8-4a4b-af45-354830d0aa45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.536734 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536420    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs\") pod \"network-metrics-daemon-ckj9h\" (UID: \"ad24b800-9783-47dc-aaa1-4d2bbc01afb8\") " pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:13.536734 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536444    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-hostroot\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.536734 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536468    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-host-slash\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.536734 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536493    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-etc-openvswitch\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.536734 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536524    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-cnibin\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.536734 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536555    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffd4833a-2f6e-4714-872b-bde81197a610-cni-binary-copy\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.536734 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536588    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-etc-kubernetes\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.536734 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536616    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-ovn-node-metrics-cert\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.537391 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536641    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c72d664d-d8e7-4a0e-a74d-516f8709e325-konnectivity-ca\") pod \"konnectivity-agent-2lhx7\" (UID: \"c72d664d-d8e7-4a0e-a74d-516f8709e325\") " pod="kube-system/konnectivity-agent-2lhx7"
Jul 06 20:49:13.537391 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536706    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-host-var-lib-cni-multus\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.537391 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536736    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/575bca76-00f8-4a4b-af45-354830d0aa45-device-dir\") pod \"aws-ebs-csi-driver-node-gscqt\" (UID: \"575bca76-00f8-4a4b-af45-354830d0aa45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.537391 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536759    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-run-systemd\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.537391 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536785    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.537391 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536828    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-host-var-lib-kubelet\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.537391 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536854    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-var-lib-openvswitch\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.537391 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536884    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.537391 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536912    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-multus-cni-dir\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.537391 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536936    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d4b9056-4c7c-43fd-a2b6-6d617f5cf503-serviceca\") pod \"node-ca-rbr6h\" (UID: \"5d4b9056-4c7c-43fd-a2b6-6d617f5cf503\") " pod="openshift-image-registry/node-ca-rbr6h"
Jul 06 20:49:13.537391 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.536982    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3ad00703-a997-4d17-b251-09093cef791f-host-slash\") pod \"iptables-alerter-5bp9d\" (UID: \"3ad00703-a997-4d17-b251-09093cef791f\") " pod="openshift-network-operator/iptables-alerter-5bp9d"
Jul 06 20:49:13.537391 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.537012    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c72d664d-d8e7-4a0e-a74d-516f8709e325-agent-certs\") pod \"konnectivity-agent-2lhx7\" (UID: \"c72d664d-d8e7-4a0e-a74d-516f8709e325\") " pod="kube-system/konnectivity-agent-2lhx7"
Jul 06 20:49:13.537391 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.537037    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-system-cni-dir\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.537391 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.537067    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-log-socket\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.537391 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.537092    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-host-cni-bin\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.537391 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.537139    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/575bca76-00f8-4a4b-af45-354830d0aa45-etc-selinux\") pod \"aws-ebs-csi-driver-node-gscqt\" (UID: \"575bca76-00f8-4a4b-af45-354830d0aa45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.538287 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.537168    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8lrq\" (UniqueName: \"kubernetes.io/projected/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-kube-api-access-p8lrq\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.538287 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.537205    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-system-cni-dir\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.538287 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.537231    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/575bca76-00f8-4a4b-af45-354830d0aa45-registration-dir\") pod \"aws-ebs-csi-driver-node-gscqt\" (UID: \"575bca76-00f8-4a4b-af45-354830d0aa45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.538287 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.537257    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-run-openvswitch\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.538287 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.537283    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-env-overrides\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.538287 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.537311    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64skr\" (UniqueName: \"kubernetes.io/projected/5d4b9056-4c7c-43fd-a2b6-6d617f5cf503-kube-api-access-64skr\") pod \"node-ca-rbr6h\" (UID: \"5d4b9056-4c7c-43fd-a2b6-6d617f5cf503\") " pod="openshift-image-registry/node-ca-rbr6h"
Jul 06 20:49:13.538287 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.537360    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.538287 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.537388    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-ovnkube-config\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.538287 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.537414    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-os-release\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.538287 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.537441    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.538287 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.537468    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-multus-socket-dir-parent\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.538287 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.537493    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-host-kubelet\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.538930 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.538438    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mzmqn"
Jul 06 20:49:13.540242 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.539997    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:13.540242 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:13.540069    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-khnhx" podUID="c260a02b-d5f8-48c9-894b-44c6310ef087"
Jul 06 20:49:13.540414 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.540364    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 20:49:13.540979 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.540955    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 06 20:49:13.541079 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.541060    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4j5j7\""
Jul 06 20:49:13.541209 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.541180    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 06 20:49:13.541493 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.541478    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.543497 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.543480    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:49:13.543794 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.543775    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 06 20:49:13.543904 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.543886    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9c8zm\""
Jul 06 20:49:13.573930 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.573909    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4fzkl"
Jul 06 20:49:13.590127 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.589839    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4fzkl"
Jul 06 20:49:13.625789 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.625763    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 06 20:49:13.638085 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638058    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-systemd-units\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.638085 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638088    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-run-ovn\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.638266 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638110    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c260a02b-d5f8-48c9-894b-44c6310ef087-kubelet-config\") pod \"global-pull-secret-syncer-khnhx\" (UID: \"c260a02b-d5f8-48c9-894b-44c6310ef087\") " pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:13.638266 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638143    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-os-release\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.638266 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638170    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zrw2\" (UniqueName: \"kubernetes.io/projected/ffd4833a-2f6e-4714-872b-bde81197a610-kube-api-access-2zrw2\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.638266 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638190    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-systemd-units\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.638266 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638217    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d4b9056-4c7c-43fd-a2b6-6d617f5cf503-host\") pod \"node-ca-rbr6h\" (UID: \"5d4b9056-4c7c-43fd-a2b6-6d617f5cf503\") " pod="openshift-image-registry/node-ca-rbr6h"
Jul 06 20:49:13.638266 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638249    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/575bca76-00f8-4a4b-af45-354830d0aa45-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gscqt\" (UID: \"575bca76-00f8-4a4b-af45-354830d0aa45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.638498 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638270    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-os-release\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.638498 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638277    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85jgm\" (UniqueName: \"kubernetes.io/projected/575bca76-00f8-4a4b-af45-354830d0aa45-kube-api-access-85jgm\") pod \"aws-ebs-csi-driver-node-gscqt\" (UID: \"575bca76-00f8-4a4b-af45-354830d0aa45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.638498 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638198    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-run-ovn\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.638498 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638312    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d4b9056-4c7c-43fd-a2b6-6d617f5cf503-host\") pod \"node-ca-rbr6h\" (UID: \"5d4b9056-4c7c-43fd-a2b6-6d617f5cf503\") " pod="openshift-image-registry/node-ca-rbr6h"
Jul 06 20:49:13.638498 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638342    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/575bca76-00f8-4a4b-af45-354830d0aa45-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gscqt\" (UID: \"575bca76-00f8-4a4b-af45-354830d0aa45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.638498 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638400    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8w9w\" (UniqueName: \"kubernetes.io/projected/3ad00703-a997-4d17-b251-09093cef791f-kube-api-access-j8w9w\") pod \"iptables-alerter-5bp9d\" (UID: \"3ad00703-a997-4d17-b251-09093cef791f\") " pod="openshift-network-operator/iptables-alerter-5bp9d"
Jul 06 20:49:13.638498 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638436    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/036a3e17-12a4-418f-b550-752a6ff9694e-hosts-file\") pod \"node-resolver-mzmqn\" (UID: \"036a3e17-12a4-418f-b550-752a6ff9694e\") " pod="openshift-dns/node-resolver-mzmqn"
Jul 06 20:49:13.638498 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638463    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h9j2\" (UniqueName: \"kubernetes.io/projected/e738125f-c5aa-41bc-9f63-2baf9c649e6b-kube-api-access-7h9j2\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.638498 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638495    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk5xx\" (UniqueName: \"kubernetes.io/projected/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-kube-api-access-jk5xx\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.638977 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638521    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-host-run-netns\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.638977 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638550    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-host-var-lib-cni-bin\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.638977 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638575    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/575bca76-00f8-4a4b-af45-354830d0aa45-sys-fs\") pod \"aws-ebs-csi-driver-node-gscqt\" (UID: \"575bca76-00f8-4a4b-af45-354830d0aa45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.638977 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638600    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-etc-kubernetes\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.638977 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638602    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-host-run-netns\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.638977 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638646    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-host-var-lib-cni-bin\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.638977 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638649    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs\") pod \"network-metrics-daemon-ckj9h\" (UID: \"ad24b800-9783-47dc-aaa1-4d2bbc01afb8\") " pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:13.638977 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638685    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/575bca76-00f8-4a4b-af45-354830d0aa45-sys-fs\") pod \"aws-ebs-csi-driver-node-gscqt\" (UID: \"575bca76-00f8-4a4b-af45-354830d0aa45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.638977 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638695    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-hostroot\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.638977 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638721    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-host-slash\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.638977 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638726    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-hostroot\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.638977 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638739    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-etc-openvswitch\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.638977 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:13.638741    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:13.638977 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638770    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-host-slash\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.638977 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638777    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-etc-openvswitch\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.638977 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638800    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-cnibin\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.638977 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:13.638816    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs podName:ad24b800-9783-47dc-aaa1-4d2bbc01afb8 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:14.138799561 +0000 UTC m=+2.973686913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs") pod "network-metrics-daemon-ckj9h" (UID: "ad24b800-9783-47dc-aaa1-4d2bbc01afb8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:13.639792 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638824    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-cnibin\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.639792 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638839    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffd4833a-2f6e-4714-872b-bde81197a610-cni-binary-copy\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.639792 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638863    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-etc-kubernetes\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.639792 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638885    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-ovn-node-metrics-cert\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.639792 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638914    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-etc-sysctl-conf\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.639792 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638939    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c72d664d-d8e7-4a0e-a74d-516f8709e325-konnectivity-ca\") pod \"konnectivity-agent-2lhx7\" (UID: \"c72d664d-d8e7-4a0e-a74d-516f8709e325\") " pod="kube-system/konnectivity-agent-2lhx7"
Jul 06 20:49:13.639792 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638963    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-host-var-lib-cni-multus\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.639792 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.638988    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/575bca76-00f8-4a4b-af45-354830d0aa45-device-dir\") pod \"aws-ebs-csi-driver-node-gscqt\" (UID: \"575bca76-00f8-4a4b-af45-354830d0aa45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.639792 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639025    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-run-systemd\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.639792 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639055    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-etc-sysctl-d\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.639792 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639053    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/575bca76-00f8-4a4b-af45-354830d0aa45-device-dir\") pod \"aws-ebs-csi-driver-node-gscqt\" (UID: \"575bca76-00f8-4a4b-af45-354830d0aa45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.639792 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639086    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.639792 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639113    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-host-var-lib-kubelet\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.639792 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639137    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-var-lib-openvswitch\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.639792 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639160    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.639792 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639186    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c260a02b-d5f8-48c9-894b-44c6310ef087-dbus\") pod \"global-pull-secret-syncer-khnhx\" (UID: \"c260a02b-d5f8-48c9-894b-44c6310ef087\") " pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:13.639792 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639208    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-etc-systemd\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.640728 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639236    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-multus-cni-dir\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.640728 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639260    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d4b9056-4c7c-43fd-a2b6-6d617f5cf503-serviceca\") pod \"node-ca-rbr6h\" (UID: \"5d4b9056-4c7c-43fd-a2b6-6d617f5cf503\") " pod="openshift-image-registry/node-ca-rbr6h"
Jul 06 20:49:13.640728 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639286    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3ad00703-a997-4d17-b251-09093cef791f-host-slash\") pod \"iptables-alerter-5bp9d\" (UID: \"3ad00703-a997-4d17-b251-09093cef791f\") " pod="openshift-network-operator/iptables-alerter-5bp9d"
Jul 06 20:49:13.640728 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639313    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c72d664d-d8e7-4a0e-a74d-516f8709e325-agent-certs\") pod \"konnectivity-agent-2lhx7\" (UID: \"c72d664d-d8e7-4a0e-a74d-516f8709e325\") " pod="kube-system/konnectivity-agent-2lhx7"
Jul 06 20:49:13.640728 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639338    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-system-cni-dir\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.640728 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639363    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-log-socket\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.640728 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639417    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-host-cni-bin\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.640728 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639437    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffd4833a-2f6e-4714-872b-bde81197a610-cni-binary-copy\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.640728 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639442    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-var-lib-kubelet\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.640728 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639465    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.640728 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639478    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-etc-kubernetes\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.640728 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639490    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-host-var-lib-cni-multus\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.640728 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639493    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/575bca76-00f8-4a4b-af45-354830d0aa45-etc-selinux\") pod \"aws-ebs-csi-driver-node-gscqt\" (UID: \"575bca76-00f8-4a4b-af45-354830d0aa45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.640728 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639543    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8lrq\" (UniqueName: \"kubernetes.io/projected/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-kube-api-access-p8lrq\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.640728 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639549    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-run-systemd\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.640728 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639561    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-host-var-lib-kubelet\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.640728 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639576    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-etc-modprobe-d\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.641536 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639590    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3ad00703-a997-4d17-b251-09093cef791f-host-slash\") pod \"iptables-alerter-5bp9d\" (UID: \"3ad00703-a997-4d17-b251-09093cef791f\") " pod="openshift-network-operator/iptables-alerter-5bp9d"
Jul 06 20:49:13.641536 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639592    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-host-cni-bin\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.641536 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639605    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-run\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.641536 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639625    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c72d664d-d8e7-4a0e-a74d-516f8709e325-konnectivity-ca\") pod \"konnectivity-agent-2lhx7\" (UID: \"c72d664d-d8e7-4a0e-a74d-516f8709e325\") " pod="kube-system/konnectivity-agent-2lhx7"
Jul 06 20:49:13.641536 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639624    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-system-cni-dir\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.641536 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639650    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-lib-modules\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.641536 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639563    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-var-lib-openvswitch\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.641536 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639551    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-log-socket\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.641536 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639685    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-system-cni-dir\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.641536 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639705    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/575bca76-00f8-4a4b-af45-354830d0aa45-etc-selinux\") pod \"aws-ebs-csi-driver-node-gscqt\" (UID: \"575bca76-00f8-4a4b-af45-354830d0aa45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.641536 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639729    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/575bca76-00f8-4a4b-af45-354830d0aa45-registration-dir\") pod \"aws-ebs-csi-driver-node-gscqt\" (UID: \"575bca76-00f8-4a4b-af45-354830d0aa45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.641536 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639762    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-run-openvswitch\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.641536 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639773    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/575bca76-00f8-4a4b-af45-354830d0aa45-registration-dir\") pod \"aws-ebs-csi-driver-node-gscqt\" (UID: \"575bca76-00f8-4a4b-af45-354830d0aa45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.641536 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639774    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-multus-cni-dir\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.641536 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639788    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-env-overrides\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.641536 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639823    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64skr\" (UniqueName: \"kubernetes.io/projected/5d4b9056-4c7c-43fd-a2b6-6d617f5cf503-kube-api-access-64skr\") pod \"node-ca-rbr6h\" (UID: \"5d4b9056-4c7c-43fd-a2b6-6d617f5cf503\") " pod="openshift-image-registry/node-ca-rbr6h"
Jul 06 20:49:13.641536 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639830    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-system-cni-dir\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.642203 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639836    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-run-openvswitch\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.642203 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639856    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.642203 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639885    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-ovnkube-config\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.642203 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639903    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.642203 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639929    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e738125f-c5aa-41bc-9f63-2baf9c649e6b-tmp\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.642203 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.639909    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 06 20:49:13.642203 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640302    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-env-overrides\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.642203 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640336    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-os-release\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.642203 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640369    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.642203 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640390    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-multus-socket-dir-parent\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.642203 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640407    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-host-kubelet\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.642203 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640425    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdzlx\" (UniqueName: \"kubernetes.io/projected/036a3e17-12a4-418f-b550-752a6ff9694e-kube-api-access-qdzlx\") pod \"node-resolver-mzmqn\" (UID: \"036a3e17-12a4-418f-b550-752a6ff9694e\") " pod="openshift-dns/node-resolver-mzmqn"
Jul 06 20:49:13.642203 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640450    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret\") pod \"global-pull-secret-syncer-khnhx\" (UID: \"c260a02b-d5f8-48c9-894b-44c6310ef087\") " pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:13.642203 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640456    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d4b9056-4c7c-43fd-a2b6-6d617f5cf503-serviceca\") pod \"node-ca-rbr6h\" (UID: \"5d4b9056-4c7c-43fd-a2b6-6d617f5cf503\") " pod="openshift-image-registry/node-ca-rbr6h"
Jul 06 20:49:13.642203 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640527    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-os-release\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.642203 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640464    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-sys\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.642203 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640561    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e738125f-c5aa-41bc-9f63-2baf9c649e6b-etc-tuned\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.642904 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640571    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.642904 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640560    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-ovnkube-config\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.642904 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640576    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.642904 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640599    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-host-kubelet\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.642904 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640622    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-multus-socket-dir-parent\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.642904 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640631    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-node-log\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.642904 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640679    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-node-log\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.642904 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640679    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.642904 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640712    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ffd4833a-2f6e-4714-872b-bde81197a610-multus-daemon-config\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.642904 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640735    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkzpl\" (UniqueName: \"kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl\") pod \"network-check-target-p9bcs\" (UID: \"806f1f1d-5cea-4e14-b4e8-a293bd4b3393\") " pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:13.642904 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640756    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-host-run-netns\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.642904 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640781    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/036a3e17-12a4-418f-b550-752a6ff9694e-tmp-dir\") pod \"node-resolver-mzmqn\" (UID: \"036a3e17-12a4-418f-b550-752a6ff9694e\") " pod="openshift-dns/node-resolver-mzmqn"
Jul 06 20:49:13.642904 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640806    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-etc-sysconfig\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.642904 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640809    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-host-run-netns\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.642904 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640829    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-host\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.642904 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.640857    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5md4t\" (UniqueName: \"kubernetes.io/projected/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-kube-api-access-5md4t\") pod \"network-metrics-daemon-ckj9h\" (UID: \"ad24b800-9783-47dc-aaa1-4d2bbc01afb8\") " pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:13.642904 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.641104    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.643709 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.641286    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ffd4833a-2f6e-4714-872b-bde81197a610-multus-daemon-config\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.643709 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.641724    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-host-run-k8s-cni-cncf-io\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.643709 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.641748    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-multus-conf-dir\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.643709 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.641772    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-host-run-multus-certs\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.643709 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.641798    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/575bca76-00f8-4a4b-af45-354830d0aa45-socket-dir\") pod \"aws-ebs-csi-driver-node-gscqt\" (UID: \"575bca76-00f8-4a4b-af45-354830d0aa45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.643709 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.641821    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-host-run-k8s-cni-cncf-io\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.643709 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.641826    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3ad00703-a997-4d17-b251-09093cef791f-iptables-alerter-script\") pod \"iptables-alerter-5bp9d\" (UID: \"3ad00703-a997-4d17-b251-09093cef791f\") " pod="openshift-network-operator/iptables-alerter-5bp9d"
Jul 06 20:49:13.643709 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.641853    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-multus-conf-dir\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.643709 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.641868    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.643709 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.641894    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-cnibin\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.643709 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.641911    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-host-run-multus-certs\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.643709 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.641920    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-host-cni-netd\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.643709 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.641948    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-ovnkube-script-lib\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.643709 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.642034    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/575bca76-00f8-4a4b-af45-354830d0aa45-socket-dir\") pod \"aws-ebs-csi-driver-node-gscqt\" (UID: \"575bca76-00f8-4a4b-af45-354830d0aa45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.643709 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.642085    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffd4833a-2f6e-4714-872b-bde81197a610-cnibin\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.643709 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.642406    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3ad00703-a997-4d17-b251-09093cef791f-iptables-alerter-script\") pod \"iptables-alerter-5bp9d\" (UID: \"3ad00703-a997-4d17-b251-09093cef791f\") " pod="openshift-network-operator/iptables-alerter-5bp9d"
Jul 06 20:49:13.643709 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.642466    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-host-cni-netd\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.644415 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.642459    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-ovnkube-script-lib\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.644415 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.642582    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.644415 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.643596    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-ovn-node-metrics-cert\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.644415 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.643753    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c72d664d-d8e7-4a0e-a74d-516f8709e325-agent-certs\") pod \"konnectivity-agent-2lhx7\" (UID: \"c72d664d-d8e7-4a0e-a74d-516f8709e325\") " pod="kube-system/konnectivity-agent-2lhx7"
Jul 06 20:49:13.653902 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.653863    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8w9w\" (UniqueName: \"kubernetes.io/projected/3ad00703-a997-4d17-b251-09093cef791f-kube-api-access-j8w9w\") pod \"iptables-alerter-5bp9d\" (UID: \"3ad00703-a997-4d17-b251-09093cef791f\") " pod="openshift-network-operator/iptables-alerter-5bp9d"
Jul 06 20:49:13.655203 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:13.655182    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:49:13.655351 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:13.655334    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:49:13.655437 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:13.655360    2559 projected.go:196] Error preparing data for projected volume kube-api-access-fkzpl for pod openshift-network-diagnostics/network-check-target-p9bcs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:13.655437 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:13.655423    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl podName:806f1f1d-5cea-4e14-b4e8-a293bd4b3393 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:14.155404283 +0000 UTC m=+2.990291647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fkzpl" (UniqueName: "kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl") pod "network-check-target-p9bcs" (UID: "806f1f1d-5cea-4e14-b4e8-a293bd4b3393") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:13.657756 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.657729    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk5xx\" (UniqueName: \"kubernetes.io/projected/a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2-kube-api-access-jk5xx\") pod \"multus-additional-cni-plugins-wf6lk\" (UID: \"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2\") " pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.658316 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.658281    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8lrq\" (UniqueName: \"kubernetes.io/projected/ec4ea812-1f43-4ea5-9f17-7f141c1b83c3-kube-api-access-p8lrq\") pod \"ovnkube-node-pt6t9\" (UID: \"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.658403 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.658327    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64skr\" (UniqueName: \"kubernetes.io/projected/5d4b9056-4c7c-43fd-a2b6-6d617f5cf503-kube-api-access-64skr\") pod \"node-ca-rbr6h\" (UID: \"5d4b9056-4c7c-43fd-a2b6-6d617f5cf503\") " pod="openshift-image-registry/node-ca-rbr6h"
Jul 06 20:49:13.658902 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.658878    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5md4t\" (UniqueName: \"kubernetes.io/projected/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-kube-api-access-5md4t\") pod \"network-metrics-daemon-ckj9h\" (UID: \"ad24b800-9783-47dc-aaa1-4d2bbc01afb8\") " pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:13.659066 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.659047    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zrw2\" (UniqueName: \"kubernetes.io/projected/ffd4833a-2f6e-4714-872b-bde81197a610-kube-api-access-2zrw2\") pod \"multus-46dgr\" (UID: \"ffd4833a-2f6e-4714-872b-bde81197a610\") " pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.659733 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.659710    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85jgm\" (UniqueName: \"kubernetes.io/projected/575bca76-00f8-4a4b-af45-354830d0aa45-kube-api-access-85jgm\") pod \"aws-ebs-csi-driver-node-gscqt\" (UID: \"575bca76-00f8-4a4b-af45-354830d0aa45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.743099 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743065    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdzlx\" (UniqueName: \"kubernetes.io/projected/036a3e17-12a4-418f-b550-752a6ff9694e-kube-api-access-qdzlx\") pod \"node-resolver-mzmqn\" (UID: \"036a3e17-12a4-418f-b550-752a6ff9694e\") " pod="openshift-dns/node-resolver-mzmqn"
Jul 06 20:49:13.743099 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743100    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret\") pod \"global-pull-secret-syncer-khnhx\" (UID: \"c260a02b-d5f8-48c9-894b-44c6310ef087\") " pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:13.743332 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743121    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-sys\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.743332 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743190    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-sys\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.743332 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:13.743242    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:13.743332 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743259    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e738125f-c5aa-41bc-9f63-2baf9c649e6b-etc-tuned\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.743332 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743337    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/036a3e17-12a4-418f-b550-752a6ff9694e-tmp-dir\") pod \"node-resolver-mzmqn\" (UID: \"036a3e17-12a4-418f-b550-752a6ff9694e\") " pod="openshift-dns/node-resolver-mzmqn"
Jul 06 20:49:13.743589 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:13.743367    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret podName:c260a02b-d5f8-48c9-894b-44c6310ef087 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:14.243335958 +0000 UTC m=+3.078223307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret") pod "global-pull-secret-syncer-khnhx" (UID: "c260a02b-d5f8-48c9-894b-44c6310ef087") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:13.743589 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743401    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-etc-sysconfig\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.743589 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743430    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-host\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.743589 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743473    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c260a02b-d5f8-48c9-894b-44c6310ef087-kubelet-config\") pod \"global-pull-secret-syncer-khnhx\" (UID: \"c260a02b-d5f8-48c9-894b-44c6310ef087\") " pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:13.743589 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743507    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/036a3e17-12a4-418f-b550-752a6ff9694e-hosts-file\") pod \"node-resolver-mzmqn\" (UID: \"036a3e17-12a4-418f-b550-752a6ff9694e\") " pod="openshift-dns/node-resolver-mzmqn"
Jul 06 20:49:13.743589 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743532    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h9j2\" (UniqueName: \"kubernetes.io/projected/e738125f-c5aa-41bc-9f63-2baf9c649e6b-kube-api-access-7h9j2\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.743589 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743541    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-etc-sysconfig\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.743589 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743562    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-etc-kubernetes\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.743589 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743568    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-host\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.743589 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743584    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c260a02b-d5f8-48c9-894b-44c6310ef087-kubelet-config\") pod \"global-pull-secret-syncer-khnhx\" (UID: \"c260a02b-d5f8-48c9-894b-44c6310ef087\") " pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:13.744221 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743606    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-etc-sysctl-conf\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.744221 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743606    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/036a3e17-12a4-418f-b550-752a6ff9694e-hosts-file\") pod \"node-resolver-mzmqn\" (UID: \"036a3e17-12a4-418f-b550-752a6ff9694e\") " pod="openshift-dns/node-resolver-mzmqn"
Jul 06 20:49:13.744221 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743666    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-etc-sysctl-d\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.744221 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743668    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-etc-kubernetes\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.744221 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743671    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/036a3e17-12a4-418f-b550-752a6ff9694e-tmp-dir\") pod \"node-resolver-mzmqn\" (UID: \"036a3e17-12a4-418f-b550-752a6ff9694e\") " pod="openshift-dns/node-resolver-mzmqn"
Jul 06 20:49:13.744221 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743698    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c260a02b-d5f8-48c9-894b-44c6310ef087-dbus\") pod \"global-pull-secret-syncer-khnhx\" (UID: \"c260a02b-d5f8-48c9-894b-44c6310ef087\") " pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:13.744221 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743756    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-etc-sysctl-conf\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.744221 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743774    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-etc-systemd\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.744221 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743809    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-etc-sysctl-d\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.744221 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743823    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-var-lib-kubelet\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.744221 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743828    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c260a02b-d5f8-48c9-894b-44c6310ef087-dbus\") pod \"global-pull-secret-syncer-khnhx\" (UID: \"c260a02b-d5f8-48c9-894b-44c6310ef087\") " pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:13.744221 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743852    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-etc-modprobe-d\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.744221 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743863    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-etc-systemd\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.744221 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743879    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-run\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.744221 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743879    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-var-lib-kubelet\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.744221 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743907    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-lib-modules\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.744221 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743947    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e738125f-c5aa-41bc-9f63-2baf9c649e6b-tmp\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.744221 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743981    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-run\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.745001 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.743986    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-etc-modprobe-d\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.745001 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.744069    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e738125f-c5aa-41bc-9f63-2baf9c649e6b-lib-modules\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.745990 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.745967    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e738125f-c5aa-41bc-9f63-2baf9c649e6b-etc-tuned\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.746184 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.746164    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e738125f-c5aa-41bc-9f63-2baf9c649e6b-tmp\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.753027 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.752961    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdzlx\" (UniqueName: \"kubernetes.io/projected/036a3e17-12a4-418f-b550-752a6ff9694e-kube-api-access-qdzlx\") pod \"node-resolver-mzmqn\" (UID: \"036a3e17-12a4-418f-b550-752a6ff9694e\") " pod="openshift-dns/node-resolver-mzmqn"
Jul 06 20:49:13.756635 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.756610    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h9j2\" (UniqueName: \"kubernetes.io/projected/e738125f-c5aa-41bc-9f63-2baf9c649e6b-kube-api-access-7h9j2\") pod \"tuned-nmhcb\" (UID: \"e738125f-c5aa-41bc-9f63-2baf9c649e6b\") " pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:13.786596 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.786569    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:49:13.838519 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.838470    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2lhx7"
Jul 06 20:49:13.848067 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.848039    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wf6lk"
Jul 06 20:49:13.857798 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.857774    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5bp9d"
Jul 06 20:49:13.864721 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.864699    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:13.871250 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.871229    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-46dgr"
Jul 06 20:49:13.879934 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.879914    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rbr6h"
Jul 06 20:49:13.888655 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.888634    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt"
Jul 06 20:49:13.896647 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.896611    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mzmqn"
Jul 06 20:49:13.902183 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:13.902165    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nmhcb"
Jul 06 20:49:14.147435 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:14.147402    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs\") pod \"network-metrics-daemon-ckj9h\" (UID: \"ad24b800-9783-47dc-aaa1-4d2bbc01afb8\") " pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:14.147656 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:14.147554    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:14.147656 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:14.147616    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs podName:ad24b800-9783-47dc-aaa1-4d2bbc01afb8 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:15.147602308 +0000 UTC m=+3.982489652 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs") pod "network-metrics-daemon-ckj9h" (UID: "ad24b800-9783-47dc-aaa1-4d2bbc01afb8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:14.247846 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:14.247812    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret\") pod \"global-pull-secret-syncer-khnhx\" (UID: \"c260a02b-d5f8-48c9-894b-44c6310ef087\") " pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:14.247846 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:14.247851    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkzpl\" (UniqueName: \"kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl\") pod \"network-check-target-p9bcs\" (UID: \"806f1f1d-5cea-4e14-b4e8-a293bd4b3393\") " pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:14.248073 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:14.247974    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:14.248073 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:14.248045    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret podName:c260a02b-d5f8-48c9-894b-44c6310ef087 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:15.248029933 +0000 UTC m=+4.082917278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret") pod "global-pull-secret-syncer-khnhx" (UID: "c260a02b-d5f8-48c9-894b-44c6310ef087") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:14.248073 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:14.248067    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:49:14.248242 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:14.248094    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:49:14.248242 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:14.248111    2559 projected.go:196] Error preparing data for projected volume kube-api-access-fkzpl for pod openshift-network-diagnostics/network-check-target-p9bcs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:14.248242 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:14.248194    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl podName:806f1f1d-5cea-4e14-b4e8-a293bd4b3393 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:15.248175288 +0000 UTC m=+4.083062649 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fkzpl" (UniqueName: "kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl") pod "network-check-target-p9bcs" (UID: "806f1f1d-5cea-4e14-b4e8-a293bd4b3393") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:14.582830 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:14.582797    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad00703_a997_4d17_b251_09093cef791f.slice/crio-d6c58f63d9b20114ed846c4f2f200497ddd619837ba2ca77473c2b70469e9f8c WatchSource:0}: Error finding container d6c58f63d9b20114ed846c4f2f200497ddd619837ba2ca77473c2b70469e9f8c: Status 404 returned error can't find the container with id d6c58f63d9b20114ed846c4f2f200497ddd619837ba2ca77473c2b70469e9f8c
Jul 06 20:49:14.584560 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:14.584529    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd4833a_2f6e_4714_872b_bde81197a610.slice/crio-6e6201950776cf6948020fe824e9e4cd7fb3a86b5fcaf37f45c3b56cc14a81c1 WatchSource:0}: Error finding container 6e6201950776cf6948020fe824e9e4cd7fb3a86b5fcaf37f45c3b56cc14a81c1: Status 404 returned error can't find the container with id 6e6201950776cf6948020fe824e9e4cd7fb3a86b5fcaf37f45c3b56cc14a81c1
Jul 06 20:49:14.584950 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:14.584926    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72d664d_d8e7_4a0e_a74d_516f8709e325.slice/crio-b9c9be647972cc3f2f73c8238e9753fee30e16ab6ed6649fc9c3a8ecb15d3c80 WatchSource:0}: Error finding container b9c9be647972cc3f2f73c8238e9753fee30e16ab6ed6649fc9c3a8ecb15d3c80: Status 404 returned error can't find the container with id b9c9be647972cc3f2f73c8238e9753fee30e16ab6ed6649fc9c3a8ecb15d3c80
Jul 06 20:49:14.589474 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:14.589450    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d4b9056_4c7c_43fd_a2b6_6d617f5cf503.slice/crio-be53b9af123f198466e47ac69e3b95c79d5795260c4e7d50185db511e5562006 WatchSource:0}: Error finding container be53b9af123f198466e47ac69e3b95c79d5795260c4e7d50185db511e5562006: Status 404 returned error can't find the container with id be53b9af123f198466e47ac69e3b95c79d5795260c4e7d50185db511e5562006
Jul 06 20:49:14.590210 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:14.590191    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode738125f_c5aa_41bc_9f63_2baf9c649e6b.slice/crio-b872939f7a4fb5a0c975a8d45c3a61d48156d295c3ef05ed6f7bd7b455e1a299 WatchSource:0}: Error finding container b872939f7a4fb5a0c975a8d45c3a61d48156d295c3ef05ed6f7bd7b455e1a299: Status 404 returned error can't find the container with id b872939f7a4fb5a0c975a8d45c3a61d48156d295c3ef05ed6f7bd7b455e1a299
Jul 06 20:49:14.591684 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:14.591613    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:44:14 +0000 UTC" deadline="2027-12-09 01:16:35.00945226 +0000 UTC"
Jul 06 20:49:14.591792 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:14.591689    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12484h27m20.417790523s"
Jul 06 20:49:14.591792 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:14.591774    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c1cb68_2ad2_4017_ba94_bfc66a42e7b2.slice/crio-66470dc76508c0c00c6e1bbc4aca8b8fb2350d55643dd9417baed47afb757fd6 WatchSource:0}: Error finding container 66470dc76508c0c00c6e1bbc4aca8b8fb2350d55643dd9417baed47afb757fd6: Status 404 returned error can't find the container with id 66470dc76508c0c00c6e1bbc4aca8b8fb2350d55643dd9417baed47afb757fd6
Jul 06 20:49:14.594235 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:14.593954    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4ea812_1f43_4ea5_9f17_7f141c1b83c3.slice/crio-4f883d953c080a8d93542b54b916e233dd18e51faf3e7f1fc8c7f7bd26a18813 WatchSource:0}: Error finding container 4f883d953c080a8d93542b54b916e233dd18e51faf3e7f1fc8c7f7bd26a18813: Status 404 returned error can't find the container with id 4f883d953c080a8d93542b54b916e233dd18e51faf3e7f1fc8c7f7bd26a18813
Jul 06 20:49:14.594753 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:14.594463    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036a3e17_12a4_418f_b550_752a6ff9694e.slice/crio-13a3f4d251f7043e7e943f4ccbdf3812d733639051cef7d5e8678a1b4b11e223 WatchSource:0}: Error finding container 13a3f4d251f7043e7e943f4ccbdf3812d733639051cef7d5e8678a1b4b11e223: Status 404 returned error can't find the container with id 13a3f4d251f7043e7e943f4ccbdf3812d733639051cef7d5e8678a1b4b11e223
Jul 06 20:49:14.595512 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:14.595011    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575bca76_00f8_4a4b_af45_354830d0aa45.slice/crio-67b207150b455d5486fea310014f5ac3af02bd6d2ae0f3527c5a213f8b7a9185 WatchSource:0}: Error finding container 67b207150b455d5486fea310014f5ac3af02bd6d2ae0f3527c5a213f8b7a9185: Status 404 returned error can't find the container with id 67b207150b455d5486fea310014f5ac3af02bd6d2ae0f3527c5a213f8b7a9185
Jul 06 20:49:14.651940 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:14.651804    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:14.652045 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:14.651821    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:14.652045 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:14.652030    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckj9h" podUID="ad24b800-9783-47dc-aaa1-4d2bbc01afb8"
Jul 06 20:49:14.652143 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:14.652099    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-khnhx" podUID="c260a02b-d5f8-48c9-894b-44c6310ef087"
Jul 06 20:49:14.659084 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:14.659060    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-46dgr" event={"ID":"ffd4833a-2f6e-4714-872b-bde81197a610","Type":"ContainerStarted","Data":"6e6201950776cf6948020fe824e9e4cd7fb3a86b5fcaf37f45c3b56cc14a81c1"}
Jul 06 20:49:14.660392 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:14.660369    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5bp9d" event={"ID":"3ad00703-a997-4d17-b251-09093cef791f","Type":"ContainerStarted","Data":"d6c58f63d9b20114ed846c4f2f200497ddd619837ba2ca77473c2b70469e9f8c"}
Jul 06 20:49:14.661549 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:14.661527    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt" event={"ID":"575bca76-00f8-4a4b-af45-354830d0aa45","Type":"ContainerStarted","Data":"67b207150b455d5486fea310014f5ac3af02bd6d2ae0f3527c5a213f8b7a9185"}
Jul 06 20:49:14.662781 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:14.662753    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rbr6h" event={"ID":"5d4b9056-4c7c-43fd-a2b6-6d617f5cf503","Type":"ContainerStarted","Data":"be53b9af123f198466e47ac69e3b95c79d5795260c4e7d50185db511e5562006"}
Jul 06 20:49:14.663907 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:14.663880    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2lhx7" event={"ID":"c72d664d-d8e7-4a0e-a74d-516f8709e325","Type":"ContainerStarted","Data":"b9c9be647972cc3f2f73c8238e9753fee30e16ab6ed6649fc9c3a8ecb15d3c80"}
Jul 06 20:49:14.664980 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:14.664953    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mzmqn" event={"ID":"036a3e17-12a4-418f-b550-752a6ff9694e","Type":"ContainerStarted","Data":"13a3f4d251f7043e7e943f4ccbdf3812d733639051cef7d5e8678a1b4b11e223"}
Jul 06 20:49:14.666123 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:14.666105    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9" event={"ID":"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3","Type":"ContainerStarted","Data":"4f883d953c080a8d93542b54b916e233dd18e51faf3e7f1fc8c7f7bd26a18813"}
Jul 06 20:49:14.667335 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:14.667312    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wf6lk" event={"ID":"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2","Type":"ContainerStarted","Data":"66470dc76508c0c00c6e1bbc4aca8b8fb2350d55643dd9417baed47afb757fd6"}
Jul 06 20:49:14.668455 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:14.668431    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nmhcb" event={"ID":"e738125f-c5aa-41bc-9f63-2baf9c649e6b","Type":"ContainerStarted","Data":"b872939f7a4fb5a0c975a8d45c3a61d48156d295c3ef05ed6f7bd7b455e1a299"}
Jul 06 20:49:15.155519 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:15.155110    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs\") pod \"network-metrics-daemon-ckj9h\" (UID: \"ad24b800-9783-47dc-aaa1-4d2bbc01afb8\") " pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:15.155519 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:15.155281    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:15.155519 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:15.155358    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs podName:ad24b800-9783-47dc-aaa1-4d2bbc01afb8 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:17.15533773 +0000 UTC m=+5.990225094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs") pod "network-metrics-daemon-ckj9h" (UID: "ad24b800-9783-47dc-aaa1-4d2bbc01afb8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:15.257131 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:15.256290    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret\") pod \"global-pull-secret-syncer-khnhx\" (UID: \"c260a02b-d5f8-48c9-894b-44c6310ef087\") " pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:15.257131 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:15.256342    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkzpl\" (UniqueName: \"kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl\") pod \"network-check-target-p9bcs\" (UID: \"806f1f1d-5cea-4e14-b4e8-a293bd4b3393\") " pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:15.257131 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:15.256486    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:49:15.257131 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:15.256505    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:49:15.257131 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:15.256518    2559 projected.go:196] Error preparing data for projected volume kube-api-access-fkzpl for pod openshift-network-diagnostics/network-check-target-p9bcs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:15.257131 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:15.256580    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl podName:806f1f1d-5cea-4e14-b4e8-a293bd4b3393 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:17.256561411 +0000 UTC m=+6.091448760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fkzpl" (UniqueName: "kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl") pod "network-check-target-p9bcs" (UID: "806f1f1d-5cea-4e14-b4e8-a293bd4b3393") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:15.257131 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:15.257031    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:15.257131 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:15.257089    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret podName:c260a02b-d5f8-48c9-894b-44c6310ef087 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:17.257071767 +0000 UTC m=+6.091959118 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret") pod "global-pull-secret-syncer-khnhx" (UID: "c260a02b-d5f8-48c9-894b-44c6310ef087") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:15.592615 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:15.592569    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:44:14 +0000 UTC" deadline="2028-01-05 20:18:18.444488909 +0000 UTC"
Jul 06 20:49:15.593099 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:15.592615    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13151h29m2.851878368s"
Jul 06 20:49:15.654420 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:15.654388    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:15.654642 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:15.654525    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9bcs" podUID="806f1f1d-5cea-4e14-b4e8-a293bd4b3393"
Jul 06 20:49:15.676595 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:15.676549    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal" event={"ID":"4b12aea9bf8b99530f8679e8c6d1d16c","Type":"ContainerStarted","Data":"1b8b26d0ef6a25cc360155466d21135a0e0e9d4b48b59e486311f0c1f849a26c"}
Jul 06 20:49:15.715920 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:15.715880    2559 generic.go:356] "Generic (PLEG): container finished" podID="3ee03c06c928493e4823bc059365a1f8" containerID="c9d5db89d42f8c56967e6b7c9db33e9f00cad79a7c39708170068849e63d05e2" exitCode=0
Jul 06 20:49:15.716108 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:15.715939    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" event={"ID":"3ee03c06c928493e4823bc059365a1f8","Type":"ContainerDied","Data":"c9d5db89d42f8c56967e6b7c9db33e9f00cad79a7c39708170068849e63d05e2"}
Jul 06 20:49:15.731908 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:15.731824    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal" podStartSLOduration=2.731803921 podStartE2EDuration="2.731803921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:15.71073762 +0000 UTC m=+4.545625011" watchObservedRunningTime="2026-07-06 20:49:15.731803921 +0000 UTC m=+4.566691290"
Jul 06 20:49:16.652254 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:16.652217    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:16.652867 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:16.652375    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckj9h" podUID="ad24b800-9783-47dc-aaa1-4d2bbc01afb8"
Jul 06 20:49:16.652867 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:16.652723    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:16.652867 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:16.652835    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-khnhx" podUID="c260a02b-d5f8-48c9-894b-44c6310ef087"
Jul 06 20:49:16.746925 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:16.746833    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" event={"ID":"3ee03c06c928493e4823bc059365a1f8","Type":"ContainerStarted","Data":"1b54694985284b4fd04f9409881a2ed8a4f38c03ebc4fe0bfc082082328d4033"}
Jul 06 20:49:17.174428 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:17.174389    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs\") pod \"network-metrics-daemon-ckj9h\" (UID: \"ad24b800-9783-47dc-aaa1-4d2bbc01afb8\") " pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:17.174630 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:17.174547    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:17.174630 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:17.174613    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs podName:ad24b800-9783-47dc-aaa1-4d2bbc01afb8 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:21.174593857 +0000 UTC m=+10.009481217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs") pod "network-metrics-daemon-ckj9h" (UID: "ad24b800-9783-47dc-aaa1-4d2bbc01afb8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:17.275931 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:17.275891    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret\") pod \"global-pull-secret-syncer-khnhx\" (UID: \"c260a02b-d5f8-48c9-894b-44c6310ef087\") " pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:17.276115 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:17.275947    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkzpl\" (UniqueName: \"kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl\") pod \"network-check-target-p9bcs\" (UID: \"806f1f1d-5cea-4e14-b4e8-a293bd4b3393\") " pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:17.276115 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:17.276094    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:49:17.276115 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:17.276113    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:49:17.276287 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:17.276128    2559 projected.go:196] Error preparing data for projected volume kube-api-access-fkzpl for pod openshift-network-diagnostics/network-check-target-p9bcs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:17.276287 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:17.276192    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl podName:806f1f1d-5cea-4e14-b4e8-a293bd4b3393 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:21.276170938 +0000 UTC m=+10.111058300 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fkzpl" (UniqueName: "kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl") pod "network-check-target-p9bcs" (UID: "806f1f1d-5cea-4e14-b4e8-a293bd4b3393") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:17.276782 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:17.276606    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:17.276782 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:17.276665    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret podName:c260a02b-d5f8-48c9-894b-44c6310ef087 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:21.276647912 +0000 UTC m=+10.111535264 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret") pod "global-pull-secret-syncer-khnhx" (UID: "c260a02b-d5f8-48c9-894b-44c6310ef087") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:17.654938 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:17.654359    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:17.654938 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:17.654494    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9bcs" podUID="806f1f1d-5cea-4e14-b4e8-a293bd4b3393"
Jul 06 20:49:18.651807 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:18.651754    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:18.652079 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:18.651894    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-khnhx" podUID="c260a02b-d5f8-48c9-894b-44c6310ef087"
Jul 06 20:49:18.652486 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:18.652277    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:18.652486 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:18.652385    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckj9h" podUID="ad24b800-9783-47dc-aaa1-4d2bbc01afb8"
Jul 06 20:49:19.652378 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:19.652347    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:19.652895 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:19.652475    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9bcs" podUID="806f1f1d-5cea-4e14-b4e8-a293bd4b3393"
Jul 06 20:49:20.652440 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:20.652390    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:20.652959 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:20.652530    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-khnhx" podUID="c260a02b-d5f8-48c9-894b-44c6310ef087"
Jul 06 20:49:20.652959 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:20.652651    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:20.652959 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:20.652782    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckj9h" podUID="ad24b800-9783-47dc-aaa1-4d2bbc01afb8"
Jul 06 20:49:21.210991 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:21.210949    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs\") pod \"network-metrics-daemon-ckj9h\" (UID: \"ad24b800-9783-47dc-aaa1-4d2bbc01afb8\") " pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:21.211170 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:21.211102    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:21.211230 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:21.211180    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs podName:ad24b800-9783-47dc-aaa1-4d2bbc01afb8 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:29.211157778 +0000 UTC m=+18.046045141 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs") pod "network-metrics-daemon-ckj9h" (UID: "ad24b800-9783-47dc-aaa1-4d2bbc01afb8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:21.311850 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:21.311805    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret\") pod \"global-pull-secret-syncer-khnhx\" (UID: \"c260a02b-d5f8-48c9-894b-44c6310ef087\") " pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:21.312044 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:21.311860    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkzpl\" (UniqueName: \"kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl\") pod \"network-check-target-p9bcs\" (UID: \"806f1f1d-5cea-4e14-b4e8-a293bd4b3393\") " pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:21.312044 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:21.312041    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:49:21.312161 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:21.312042    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:21.312161 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:21.312061    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:49:21.312161 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:21.312075    2559 projected.go:196] Error preparing data for projected volume kube-api-access-fkzpl for pod openshift-network-diagnostics/network-check-target-p9bcs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:21.312161 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:21.312127    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret podName:c260a02b-d5f8-48c9-894b-44c6310ef087 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:29.312105725 +0000 UTC m=+18.146993077 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret") pod "global-pull-secret-syncer-khnhx" (UID: "c260a02b-d5f8-48c9-894b-44c6310ef087") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:21.312161 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:21.312149    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl podName:806f1f1d-5cea-4e14-b4e8-a293bd4b3393 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:29.31213873 +0000 UTC m=+18.147026078 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fkzpl" (UniqueName: "kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl") pod "network-check-target-p9bcs" (UID: "806f1f1d-5cea-4e14-b4e8-a293bd4b3393") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:21.653335 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:21.652855    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:21.653335 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:21.652973    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9bcs" podUID="806f1f1d-5cea-4e14-b4e8-a293bd4b3393"
Jul 06 20:49:22.653009 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:22.652413    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:22.653009 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:22.652563    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-khnhx" podUID="c260a02b-d5f8-48c9-894b-44c6310ef087"
Jul 06 20:49:22.653009 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:22.652833    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:22.653009 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:22.652973    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckj9h" podUID="ad24b800-9783-47dc-aaa1-4d2bbc01afb8"
Jul 06 20:49:23.652967 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:23.652822    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:23.652967 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:23.652956    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9bcs" podUID="806f1f1d-5cea-4e14-b4e8-a293bd4b3393"
Jul 06 20:49:24.651571 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:24.651531    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:24.651770 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:24.651547    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:24.651770 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:24.651696    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckj9h" podUID="ad24b800-9783-47dc-aaa1-4d2bbc01afb8"
Jul 06 20:49:24.651867 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:24.651824    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-khnhx" podUID="c260a02b-d5f8-48c9-894b-44c6310ef087"
Jul 06 20:49:25.652463 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:25.652436    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:25.652897 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:25.652555    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9bcs" podUID="806f1f1d-5cea-4e14-b4e8-a293bd4b3393"
Jul 06 20:49:26.651985 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:26.651945    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:26.652166 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:26.652071    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckj9h" podUID="ad24b800-9783-47dc-aaa1-4d2bbc01afb8"
Jul 06 20:49:26.652166 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:26.652136    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:26.652276 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:26.652242    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-khnhx" podUID="c260a02b-d5f8-48c9-894b-44c6310ef087"
Jul 06 20:49:27.652058 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:27.652024    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:27.652490 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:27.652129    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9bcs" podUID="806f1f1d-5cea-4e14-b4e8-a293bd4b3393"
Jul 06 20:49:28.652345 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:28.652309    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:28.652852 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:28.652358    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:28.652852 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:28.652447    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-khnhx" podUID="c260a02b-d5f8-48c9-894b-44c6310ef087"
Jul 06 20:49:28.652852 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:28.652569    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckj9h" podUID="ad24b800-9783-47dc-aaa1-4d2bbc01afb8"
Jul 06 20:49:29.274311 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:29.274272    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs\") pod \"network-metrics-daemon-ckj9h\" (UID: \"ad24b800-9783-47dc-aaa1-4d2bbc01afb8\") " pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:29.274501 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:29.274444    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:29.274569 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:29.274518    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs podName:ad24b800-9783-47dc-aaa1-4d2bbc01afb8 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:45.274501237 +0000 UTC m=+34.109388596 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs") pod "network-metrics-daemon-ckj9h" (UID: "ad24b800-9783-47dc-aaa1-4d2bbc01afb8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:29.374835 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:29.374801    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret\") pod \"global-pull-secret-syncer-khnhx\" (UID: \"c260a02b-d5f8-48c9-894b-44c6310ef087\") " pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:29.374835 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:29.374836    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkzpl\" (UniqueName: \"kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl\") pod \"network-check-target-p9bcs\" (UID: \"806f1f1d-5cea-4e14-b4e8-a293bd4b3393\") " pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:29.375062 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:29.374949    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:49:29.375062 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:29.374975    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:49:29.375062 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:29.374988    2559 projected.go:196] Error preparing data for projected volume kube-api-access-fkzpl for pod openshift-network-diagnostics/network-check-target-p9bcs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:29.375062 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:29.375032    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl podName:806f1f1d-5cea-4e14-b4e8-a293bd4b3393 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:45.375019046 +0000 UTC m=+34.209906390 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fkzpl" (UniqueName: "kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl") pod "network-check-target-p9bcs" (UID: "806f1f1d-5cea-4e14-b4e8-a293bd4b3393") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:29.375062 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:29.374946    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:29.375266 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:29.375118    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret podName:c260a02b-d5f8-48c9-894b-44c6310ef087 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:45.375098016 +0000 UTC m=+34.209985365 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret") pod "global-pull-secret-syncer-khnhx" (UID: "c260a02b-d5f8-48c9-894b-44c6310ef087") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:29.651778 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:29.651752    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:29.651932 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:29.651883    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9bcs" podUID="806f1f1d-5cea-4e14-b4e8-a293bd4b3393"
Jul 06 20:49:30.651763 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:30.651732    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:30.652176 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:30.651732    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:30.652176 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:30.651848    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-khnhx" podUID="c260a02b-d5f8-48c9-894b-44c6310ef087"
Jul 06 20:49:30.652176 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:30.651955    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckj9h" podUID="ad24b800-9783-47dc-aaa1-4d2bbc01afb8"
Jul 06 20:49:31.652483 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:31.652439    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:31.652946 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:31.652545    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9bcs" podUID="806f1f1d-5cea-4e14-b4e8-a293bd4b3393"
Jul 06 20:49:32.651479 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:32.651444    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:32.651681 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:32.651444    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:32.651681 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:32.651566    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-khnhx" podUID="c260a02b-d5f8-48c9-894b-44c6310ef087"
Jul 06 20:49:32.651811 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:32.651677    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckj9h" podUID="ad24b800-9783-47dc-aaa1-4d2bbc01afb8"
Jul 06 20:49:33.652305 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:33.652270    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:33.652920 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:33.652390    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9bcs" podUID="806f1f1d-5cea-4e14-b4e8-a293bd4b3393"
Jul 06 20:49:34.651871 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:34.651493    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:34.652038 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:34.651558    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:34.652038 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:34.651988    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckj9h" podUID="ad24b800-9783-47dc-aaa1-4d2bbc01afb8"
Jul 06 20:49:34.652147 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:34.652045    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-khnhx" podUID="c260a02b-d5f8-48c9-894b-44c6310ef087"
Jul 06 20:49:34.787987 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:34.787953    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-46dgr" event={"ID":"ffd4833a-2f6e-4714-872b-bde81197a610","Type":"ContainerStarted","Data":"18cb19bf41c2b9886c24c3a11fa87b0b3f3b1fcd52dec60ed83082bc369b61fd"}
Jul 06 20:49:34.789696 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:34.789672    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt" event={"ID":"575bca76-00f8-4a4b-af45-354830d0aa45","Type":"ContainerStarted","Data":"51f59b422b6a15bbb8a7e5c38289062e327227d514782970bd986935bd58099f"}
Jul 06 20:49:34.793326 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:34.793306    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rbr6h" event={"ID":"5d4b9056-4c7c-43fd-a2b6-6d617f5cf503","Type":"ContainerStarted","Data":"8b63e32b7e793cbc3e47a93349dc39584d193d2fb3caab556ee94f9c035054e9"}
Jul 06 20:49:34.794798 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:34.794769    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2lhx7" event={"ID":"c72d664d-d8e7-4a0e-a74d-516f8709e325","Type":"ContainerStarted","Data":"3ba63f715e748baaaca1ce4bfa4d1ded4a4f99442fcb321b29565789308bc503"}
Jul 06 20:49:34.796194 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:34.796172    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mzmqn" event={"ID":"036a3e17-12a4-418f-b550-752a6ff9694e","Type":"ContainerStarted","Data":"fb6b6607de7beb2916c361794f6446a70383772d65e79aa8c4725fa4d5ae1622"}
Jul 06 20:49:34.799228 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:34.799208    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9" event={"ID":"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3","Type":"ContainerStarted","Data":"635f28aacdbfd6c9c1ffb82ca148f1f587182ba54569addb622e551c44613e87"}
Jul 06 20:49:34.799324 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:34.799277    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9" event={"ID":"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3","Type":"ContainerStarted","Data":"c65e968ea9a64f2ff0fdaecb8edd5906be6af2639db96041442e307ad0672185"}
Jul 06 20:49:34.799380 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:34.799328    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9" event={"ID":"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3","Type":"ContainerStarted","Data":"eda06ab3df8770170f8ce848a8a53e644fda28940803e15cc4d0320c69869522"}
Jul 06 20:49:34.799431 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:34.799379    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9" event={"ID":"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3","Type":"ContainerStarted","Data":"3f40ffe6ba5765d65e488a81222de63704e57bfd18f5cc4b45e491873277cc60"}
Jul 06 20:49:34.799431 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:34.799425    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9" event={"ID":"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3","Type":"ContainerStarted","Data":"54ce20a7b6285f1b055e431f11ece139ec47b386047d2e07064bfcf84a7291b7"}
Jul 06 20:49:34.799525 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:34.799472    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9" event={"ID":"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3","Type":"ContainerStarted","Data":"bd1aae34df11f8d5656050043e81a2fb4132759648c8c6ef7867a5e4e2126d00"}
Jul 06 20:49:34.800822 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:34.800803    2559 generic.go:356] "Generic (PLEG): container finished" podID="a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2" containerID="1b60193d576ee97b1a8bd9d364fc8d9b3b9573087bee1ba76a9cc816c8c9289b" exitCode=0
Jul 06 20:49:34.800910 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:34.800849    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wf6lk" event={"ID":"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2","Type":"ContainerDied","Data":"1b60193d576ee97b1a8bd9d364fc8d9b3b9573087bee1ba76a9cc816c8c9289b"}
Jul 06 20:49:34.802679 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:34.802656    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nmhcb" event={"ID":"e738125f-c5aa-41bc-9f63-2baf9c649e6b","Type":"ContainerStarted","Data":"d5f77687950de4f4af6761ab31d88b8f2185639d004a0469da955059d7d8c800"}
Jul 06 20:49:34.807172 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:34.807118    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" podStartSLOduration=21.807086495 podStartE2EDuration="21.807086495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:16.767064411 +0000 UTC m=+5.601951783" watchObservedRunningTime="2026-07-06 20:49:34.807086495 +0000 UTC m=+23.641973854"
Jul 06 20:49:34.807694 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:34.807661    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-46dgr" podStartSLOduration=3.526873904 podStartE2EDuration="22.807648656s" totalImagesPullingTime="19.280774752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:34.8072713 +0000 UTC m=+23.642158677" watchObservedRunningTime="2026-07-06 20:49:34.807648656 +0000 UTC m=+23.642536022"
Jul 06 20:49:34.822315 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:34.822278    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mzmqn" podStartSLOduration=3.602430161 podStartE2EDuration="22.822266298s" totalImagesPullingTime="19.219836137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:34.821735101 +0000 UTC m=+23.656622490" watchObservedRunningTime="2026-07-06 20:49:34.822266298 +0000 UTC m=+23.657153663"
Jul 06 20:49:34.838944 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:34.838895    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nmhcb" podStartSLOduration=2.615940557 podStartE2EDuration="21.838884098s" totalImagesPullingTime="19.222943541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:34.838852553 +0000 UTC m=+23.673739918" watchObservedRunningTime="2026-07-06 20:49:34.838884098 +0000 UTC m=+23.673771462"
Jul 06 20:49:34.853074 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:34.853032    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2lhx7" podStartSLOduration=3.625853438 podStartE2EDuration="22.853016824s" totalImagesPullingTime="19.227163386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:34.852793914 +0000 UTC m=+23.687681279" watchObservedRunningTime="2026-07-06 20:49:34.853016824 +0000 UTC m=+23.687904185"
Jul 06 20:49:34.960369 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:34.960338    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2lhx7_c72d664d-d8e7-4a0e-a74d-516f8709e325/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:35.335219 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:35.335197    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 06 20:49:35.606668 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:35.606498    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T20:49:35.335214855Z","UUID":"8de1543b-6bcc-4cb0-9213-1bffe0877ed0","Handler":null,"Name":"","Endpoint":""}
Jul 06 20:49:35.608132 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:35.608115    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 06 20:49:35.608238 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:35.608146    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 06 20:49:35.652151 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:35.652127    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:35.652310 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:35.652239    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9bcs" podUID="806f1f1d-5cea-4e14-b4e8-a293bd4b3393"
Jul 06 20:49:35.807245 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:35.807207    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt" event={"ID":"575bca76-00f8-4a4b-af45-354830d0aa45","Type":"ContainerStarted","Data":"837b1d679793739b6ff80534d2745d467b7cde8ee71ca198c91b095ea0d53928"}
Jul 06 20:49:35.808769 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:35.808740    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5bp9d" event={"ID":"3ad00703-a997-4d17-b251-09093cef791f","Type":"ContainerStarted","Data":"812e42458111e137e4e4991e4bdd0ecb9b8ebaa45c0002fb19bb5563b153a6df"}
Jul 06 20:49:35.824995 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:35.824935    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5bp9d" podStartSLOduration=4.586966408 podStartE2EDuration="23.824916919s" totalImagesPullingTime="19.237950511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:35.82483098 +0000 UTC m=+24.659718366" watchObservedRunningTime="2026-07-06 20:49:35.824916919 +0000 UTC m=+24.659804286"
Jul 06 20:49:35.825416 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:35.825383    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rbr6h" podStartSLOduration=4.602426512 podStartE2EDuration="23.825373227s" totalImagesPullingTime="19.222946715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:34.889292336 +0000 UTC m=+23.724179711" watchObservedRunningTime="2026-07-06 20:49:35.825373227 +0000 UTC m=+24.660260591"
Jul 06 20:49:36.062765 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:36.062728    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2lhx7_c72d664d-d8e7-4a0e-a74d-516f8709e325/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:36.593071 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:36.593045    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2lhx7"
Jul 06 20:49:36.593678 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:36.593656    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2lhx7"
Jul 06 20:49:36.651548 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:36.651523    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:36.651704 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:36.651673    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-khnhx" podUID="c260a02b-d5f8-48c9-894b-44c6310ef087"
Jul 06 20:49:36.651814 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:36.651718    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:36.651876 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:36.651819    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckj9h" podUID="ad24b800-9783-47dc-aaa1-4d2bbc01afb8"
Jul 06 20:49:36.814792 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:36.814523    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9" event={"ID":"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3","Type":"ContainerStarted","Data":"467ae39b327cfe9ce5f61d121c469a8f112a212f6a216bb93999afde37ac45de"}
Jul 06 20:49:36.817205 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:36.817163    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt" event={"ID":"575bca76-00f8-4a4b-af45-354830d0aa45","Type":"ContainerStarted","Data":"0e9cae972d16d2692bd931dc9e56f6e052283743c8ea97c3e978d63e98403159"}
Jul 06 20:49:36.817505 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:36.817486    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2lhx7"
Jul 06 20:49:36.818015 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:36.817997    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2lhx7"
Jul 06 20:49:36.839883 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:36.838281    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gscqt" podStartSLOduration=3.210418104 podStartE2EDuration="24.838262929s" totalImagesPullingTime="21.627844825s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:36.836863051 +0000 UTC m=+25.671750437" watchObservedRunningTime="2026-07-06 20:49:36.838262929 +0000 UTC m=+25.673150294"
Jul 06 20:49:36.902280 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:36.902248    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2lhx7_c72d664d-d8e7-4a0e-a74d-516f8709e325/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:37.655104 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:37.655076    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:37.655273 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:37.655167    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9bcs" podUID="806f1f1d-5cea-4e14-b4e8-a293bd4b3393"
Jul 06 20:49:38.652326 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:38.652295    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:38.652853 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:38.652295    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:38.652853 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:38.652421    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckj9h" podUID="ad24b800-9783-47dc-aaa1-4d2bbc01afb8"
Jul 06 20:49:38.652853 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:38.652508    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-khnhx" podUID="c260a02b-d5f8-48c9-894b-44c6310ef087"
Jul 06 20:49:39.652133 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:39.651954    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:39.652273 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:39.652207    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9bcs" podUID="806f1f1d-5cea-4e14-b4e8-a293bd4b3393"
Jul 06 20:49:39.828445 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:39.828410    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9" event={"ID":"ec4ea812-1f43-4ea5-9f17-7f141c1b83c3","Type":"ContainerStarted","Data":"63f21b787282127f733a9ea6838b5127aa01e9bb6e9052818861bc0cbf490232"}
Jul 06 20:49:39.829215 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:39.828685    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:39.830292 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:39.830270    2559 generic.go:356] "Generic (PLEG): container finished" podID="a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2" containerID="17aa88de64199a96180dfebe54d4f81b348eb72d7f46d9480c94e416c4d245ff" exitCode=0
Jul 06 20:49:39.830404 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:39.830307    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wf6lk" event={"ID":"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2","Type":"ContainerDied","Data":"17aa88de64199a96180dfebe54d4f81b348eb72d7f46d9480c94e416c4d245ff"}
Jul 06 20:49:39.844795 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:39.844769    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:39.860867 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:39.860823    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9" podStartSLOduration=8.566175081 podStartE2EDuration="27.860809339s" totalImagesPullingTime="19.294634258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:39.860305255 +0000 UTC m=+28.695192639" watchObservedRunningTime="2026-07-06 20:49:39.860809339 +0000 UTC m=+28.695696700"
Jul 06 20:49:40.404904 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:40.404870    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2lhx7_c72d664d-d8e7-4a0e-a74d-516f8709e325/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:40.505992 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:40.505960    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2lhx7_c72d664d-d8e7-4a0e-a74d-516f8709e325/konnectivity-agent/0.log" limit=1024
Jul 06 20:49:40.651921 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:40.651885    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:40.652065 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:40.651885    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:40.652065 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:40.652021    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-khnhx" podUID="c260a02b-d5f8-48c9-894b-44c6310ef087"
Jul 06 20:49:40.652162 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:40.652064    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckj9h" podUID="ad24b800-9783-47dc-aaa1-4d2bbc01afb8"
Jul 06 20:49:40.834669 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:40.834638    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wf6lk" event={"ID":"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2","Type":"ContainerStarted","Data":"6713a1f334fb4c56c225b966f77cb2a12b5eadf4a41b99184312d553027d006a"}
Jul 06 20:49:40.835270 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:40.835250    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:40.835381 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:40.835326    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:40.850515 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:40.850491    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:49:41.330461 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:41.330222    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p9bcs"]
Jul 06 20:49:41.330669 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:41.330568    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:41.330742 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:41.330678    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9bcs" podUID="806f1f1d-5cea-4e14-b4e8-a293bd4b3393"
Jul 06 20:49:41.333287 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:41.333257    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-khnhx"]
Jul 06 20:49:41.333407 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:41.333387    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:41.333568 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:41.333489    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-khnhx" podUID="c260a02b-d5f8-48c9-894b-44c6310ef087"
Jul 06 20:49:41.333958 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:41.333938    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ckj9h"]
Jul 06 20:49:41.334049 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:41.334020    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:41.334113 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:41.334098    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckj9h" podUID="ad24b800-9783-47dc-aaa1-4d2bbc01afb8"
Jul 06 20:49:41.839299 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:41.839268    2559 generic.go:356] "Generic (PLEG): container finished" podID="a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2" containerID="6713a1f334fb4c56c225b966f77cb2a12b5eadf4a41b99184312d553027d006a" exitCode=0
Jul 06 20:49:41.839737 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:41.839365    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wf6lk" event={"ID":"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2","Type":"ContainerDied","Data":"6713a1f334fb4c56c225b966f77cb2a12b5eadf4a41b99184312d553027d006a"}
Jul 06 20:49:42.652110 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:42.652083    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:42.652276 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:42.652216    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckj9h" podUID="ad24b800-9783-47dc-aaa1-4d2bbc01afb8"
Jul 06 20:49:42.652343 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:42.652270    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:42.652421 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:42.652396    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9bcs" podUID="806f1f1d-5cea-4e14-b4e8-a293bd4b3393"
Jul 06 20:49:43.652017 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:43.651978    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:43.652770 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:43.652116    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-khnhx" podUID="c260a02b-d5f8-48c9-894b-44c6310ef087"
Jul 06 20:49:43.845473 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:43.845436    2559 generic.go:356] "Generic (PLEG): container finished" podID="a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2" containerID="3f9d96e38ed2935ae54d0c4da1b2a58eaf41582e9514b4c3d412799a834e429a" exitCode=0
Jul 06 20:49:43.845642 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:43.845503    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wf6lk" event={"ID":"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2","Type":"ContainerDied","Data":"3f9d96e38ed2935ae54d0c4da1b2a58eaf41582e9514b4c3d412799a834e429a"}
Jul 06 20:49:44.651745 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:44.651706    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:44.651745 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:44.651742    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:44.651982 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:44.651830    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9bcs" podUID="806f1f1d-5cea-4e14-b4e8-a293bd4b3393"
Jul 06 20:49:44.651982 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:44.651966    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckj9h" podUID="ad24b800-9783-47dc-aaa1-4d2bbc01afb8"
Jul 06 20:49:45.292894 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:45.292856    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs\") pod \"network-metrics-daemon-ckj9h\" (UID: \"ad24b800-9783-47dc-aaa1-4d2bbc01afb8\") " pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:45.293321 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:45.292982    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:45.293321 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:45.293050    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs podName:ad24b800-9783-47dc-aaa1-4d2bbc01afb8 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:17.293031432 +0000 UTC m=+66.127918777 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs") pod "network-metrics-daemon-ckj9h" (UID: "ad24b800-9783-47dc-aaa1-4d2bbc01afb8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:49:45.393943 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:45.393907    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret\") pod \"global-pull-secret-syncer-khnhx\" (UID: \"c260a02b-d5f8-48c9-894b-44c6310ef087\") " pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:45.393943 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:45.393957    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkzpl\" (UniqueName: \"kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl\") pod \"network-check-target-p9bcs\" (UID: \"806f1f1d-5cea-4e14-b4e8-a293bd4b3393\") " pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:45.394167 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:45.394075    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:45.394167 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:45.394087    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:49:45.394167 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:45.394105    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:49:45.394167 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:45.394117    2559 projected.go:196] Error preparing data for projected volume kube-api-access-fkzpl for pod openshift-network-diagnostics/network-check-target-p9bcs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:45.394167 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:45.394152    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret podName:c260a02b-d5f8-48c9-894b-44c6310ef087 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:17.394132906 +0000 UTC m=+66.229020257 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret") pod "global-pull-secret-syncer-khnhx" (UID: "c260a02b-d5f8-48c9-894b-44c6310ef087") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:49:45.394407 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:45.394172    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl podName:806f1f1d-5cea-4e14-b4e8-a293bd4b3393 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:17.394163487 +0000 UTC m=+66.229050835 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fkzpl" (UniqueName: "kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl") pod "network-check-target-p9bcs" (UID: "806f1f1d-5cea-4e14-b4e8-a293bd4b3393") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:49:45.652232 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:45.652203    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:45.652412 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:45.652328    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-khnhx" podUID="c260a02b-d5f8-48c9-894b-44c6310ef087"
Jul 06 20:49:46.651514 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:46.651473    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:46.652035 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:46.651528    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:46.652035 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:46.651629    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9bcs" podUID="806f1f1d-5cea-4e14-b4e8-a293bd4b3393"
Jul 06 20:49:46.652035 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:46.651754    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckj9h" podUID="ad24b800-9783-47dc-aaa1-4d2bbc01afb8"
Jul 06 20:49:47.473130 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.473102    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeReady"
Jul 06 20:49:47.473310 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.473242    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 06 20:49:47.532489 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.532456    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nm7gc"]
Jul 06 20:49:47.537359 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.537324    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dhdnl"]
Jul 06 20:49:47.537662 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.537638    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nm7gc"
Jul 06 20:49:47.540470 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.540446    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dhdnl"
Jul 06 20:49:47.540993 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.540783    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 06 20:49:47.540993 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.540796    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cpffn\""
Jul 06 20:49:47.540993 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.540927    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 06 20:49:47.544603 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.544551    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 06 20:49:47.545277 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.544824    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 06 20:49:47.545277 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.545150    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zzls9\""
Jul 06 20:49:47.545908 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.545712    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 06 20:49:47.546027 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.546003    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dhdnl"]
Jul 06 20:49:47.553699 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.550569    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nm7gc"]
Jul 06 20:49:47.608853 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.608773    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c6274f2-a3f1-4130-972b-db17b036e580-config-volume\") pod \"dns-default-nm7gc\" (UID: \"7c6274f2-a3f1-4130-972b-db17b036e580\") " pod="openshift-dns/dns-default-nm7gc"
Jul 06 20:49:47.608853 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.608814    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zrm4\" (UniqueName: \"kubernetes.io/projected/7c6274f2-a3f1-4130-972b-db17b036e580-kube-api-access-8zrm4\") pod \"dns-default-nm7gc\" (UID: \"7c6274f2-a3f1-4130-972b-db17b036e580\") " pod="openshift-dns/dns-default-nm7gc"
Jul 06 20:49:47.609054 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.608861    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c6274f2-a3f1-4130-972b-db17b036e580-tmp-dir\") pod \"dns-default-nm7gc\" (UID: \"7c6274f2-a3f1-4130-972b-db17b036e580\") " pod="openshift-dns/dns-default-nm7gc"
Jul 06 20:49:47.609054 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.608891    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c6274f2-a3f1-4130-972b-db17b036e580-metrics-tls\") pod \"dns-default-nm7gc\" (UID: \"7c6274f2-a3f1-4130-972b-db17b036e580\") " pod="openshift-dns/dns-default-nm7gc"
Jul 06 20:49:47.652276 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.652242    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:49:47.660647 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.660592    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 20:49:47.710121 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.710068    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8hd8\" (UniqueName: \"kubernetes.io/projected/6524b812-b7e8-440c-891d-c219ad395486-kube-api-access-k8hd8\") pod \"ingress-canary-dhdnl\" (UID: \"6524b812-b7e8-440c-891d-c219ad395486\") " pod="openshift-ingress-canary/ingress-canary-dhdnl"
Jul 06 20:49:47.710121 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.710125    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c6274f2-a3f1-4130-972b-db17b036e580-config-volume\") pod \"dns-default-nm7gc\" (UID: \"7c6274f2-a3f1-4130-972b-db17b036e580\") " pod="openshift-dns/dns-default-nm7gc"
Jul 06 20:49:47.710368 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.710160    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zrm4\" (UniqueName: \"kubernetes.io/projected/7c6274f2-a3f1-4130-972b-db17b036e580-kube-api-access-8zrm4\") pod \"dns-default-nm7gc\" (UID: \"7c6274f2-a3f1-4130-972b-db17b036e580\") " pod="openshift-dns/dns-default-nm7gc"
Jul 06 20:49:47.710368 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.710194    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6524b812-b7e8-440c-891d-c219ad395486-cert\") pod \"ingress-canary-dhdnl\" (UID: \"6524b812-b7e8-440c-891d-c219ad395486\") " pod="openshift-ingress-canary/ingress-canary-dhdnl"
Jul 06 20:49:47.710368 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.710233    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c6274f2-a3f1-4130-972b-db17b036e580-tmp-dir\") pod \"dns-default-nm7gc\" (UID: \"7c6274f2-a3f1-4130-972b-db17b036e580\") " pod="openshift-dns/dns-default-nm7gc"
Jul 06 20:49:47.710368 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.710261    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c6274f2-a3f1-4130-972b-db17b036e580-metrics-tls\") pod \"dns-default-nm7gc\" (UID: \"7c6274f2-a3f1-4130-972b-db17b036e580\") " pod="openshift-dns/dns-default-nm7gc"
Jul 06 20:49:47.710567 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:47.710389    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:49:47.710567 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:47.710451    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c6274f2-a3f1-4130-972b-db17b036e580-metrics-tls podName:7c6274f2-a3f1-4130-972b-db17b036e580 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:48.210430903 +0000 UTC m=+37.045318246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c6274f2-a3f1-4130-972b-db17b036e580-metrics-tls") pod "dns-default-nm7gc" (UID: "7c6274f2-a3f1-4130-972b-db17b036e580") : secret "dns-default-metrics-tls" not found
Jul 06 20:49:47.710567 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.710552    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c6274f2-a3f1-4130-972b-db17b036e580-tmp-dir\") pod \"dns-default-nm7gc\" (UID: \"7c6274f2-a3f1-4130-972b-db17b036e580\") " pod="openshift-dns/dns-default-nm7gc"
Jul 06 20:49:47.710855 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.710831    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c6274f2-a3f1-4130-972b-db17b036e580-config-volume\") pod \"dns-default-nm7gc\" (UID: \"7c6274f2-a3f1-4130-972b-db17b036e580\") " pod="openshift-dns/dns-default-nm7gc"
Jul 06 20:49:47.724015 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.723826    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zrm4\" (UniqueName: \"kubernetes.io/projected/7c6274f2-a3f1-4130-972b-db17b036e580-kube-api-access-8zrm4\") pod \"dns-default-nm7gc\" (UID: \"7c6274f2-a3f1-4130-972b-db17b036e580\") " pod="openshift-dns/dns-default-nm7gc"
Jul 06 20:49:47.810836 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.810798    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8hd8\" (UniqueName: \"kubernetes.io/projected/6524b812-b7e8-440c-891d-c219ad395486-kube-api-access-k8hd8\") pod \"ingress-canary-dhdnl\" (UID: \"6524b812-b7e8-440c-891d-c219ad395486\") " pod="openshift-ingress-canary/ingress-canary-dhdnl"
Jul 06 20:49:47.811029 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.810854    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6524b812-b7e8-440c-891d-c219ad395486-cert\") pod \"ingress-canary-dhdnl\" (UID: \"6524b812-b7e8-440c-891d-c219ad395486\") " pod="openshift-ingress-canary/ingress-canary-dhdnl"
Jul 06 20:49:47.813656 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.813606    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6524b812-b7e8-440c-891d-c219ad395486-cert\") pod \"ingress-canary-dhdnl\" (UID: \"6524b812-b7e8-440c-891d-c219ad395486\") " pod="openshift-ingress-canary/ingress-canary-dhdnl"
Jul 06 20:49:47.820091 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.820063    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8hd8\" (UniqueName: \"kubernetes.io/projected/6524b812-b7e8-440c-891d-c219ad395486-kube-api-access-k8hd8\") pod \"ingress-canary-dhdnl\" (UID: \"6524b812-b7e8-440c-891d-c219ad395486\") " pod="openshift-ingress-canary/ingress-canary-dhdnl"
Jul 06 20:49:47.860735 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:47.860629    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dhdnl"
Jul 06 20:49:48.199130 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:48.199085    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dhdnl"]
Jul 06 20:49:48.212644 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:49:48.212601    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6524b812_b7e8_440c_891d_c219ad395486.slice/crio-20ee8e2478d71f18d8188420890a2310f7ad45545e90f17140299301ea2ba5c7 WatchSource:0}: Error finding container 20ee8e2478d71f18d8188420890a2310f7ad45545e90f17140299301ea2ba5c7: Status 404 returned error can't find the container with id 20ee8e2478d71f18d8188420890a2310f7ad45545e90f17140299301ea2ba5c7
Jul 06 20:49:48.212842 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:48.212827    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c6274f2-a3f1-4130-972b-db17b036e580-metrics-tls\") pod \"dns-default-nm7gc\" (UID: \"7c6274f2-a3f1-4130-972b-db17b036e580\") " pod="openshift-dns/dns-default-nm7gc"
Jul 06 20:49:48.212979 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:48.212960    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:49:48.213032 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:48.213022    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c6274f2-a3f1-4130-972b-db17b036e580-metrics-tls podName:7c6274f2-a3f1-4130-972b-db17b036e580 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:49.213004424 +0000 UTC m=+38.047891787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c6274f2-a3f1-4130-972b-db17b036e580-metrics-tls") pod "dns-default-nm7gc" (UID: "7c6274f2-a3f1-4130-972b-db17b036e580") : secret "dns-default-metrics-tls" not found
Jul 06 20:49:48.652206 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:48.652173    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:49:48.652393 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:48.652183    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:49:48.654794 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:48.654766    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 20:49:48.655570 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:48.655508    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 20:49:48.655570 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:48.655527    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hgfwv\""
Jul 06 20:49:48.655570 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:48.655554    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 20:49:48.655802 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:48.655592    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mf9wp\""
Jul 06 20:49:48.859903 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:48.859864    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dhdnl" event={"ID":"6524b812-b7e8-440c-891d-c219ad395486","Type":"ContainerStarted","Data":"20ee8e2478d71f18d8188420890a2310f7ad45545e90f17140299301ea2ba5c7"}
Jul 06 20:49:48.862986 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:48.862954    2559 generic.go:356] "Generic (PLEG): container finished" podID="a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2" containerID="710275ef30929afa7180da65530fe21809e233844f7acce25d4584da01fb7d29" exitCode=0
Jul 06 20:49:48.863124 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:48.863003    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wf6lk" event={"ID":"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2","Type":"ContainerDied","Data":"710275ef30929afa7180da65530fe21809e233844f7acce25d4584da01fb7d29"}
Jul 06 20:49:49.220771 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:49.220628    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c6274f2-a3f1-4130-972b-db17b036e580-metrics-tls\") pod \"dns-default-nm7gc\" (UID: \"7c6274f2-a3f1-4130-972b-db17b036e580\") " pod="openshift-dns/dns-default-nm7gc"
Jul 06 20:49:49.220951 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:49.220814    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:49:49.220951 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:49.220892    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c6274f2-a3f1-4130-972b-db17b036e580-metrics-tls podName:7c6274f2-a3f1-4130-972b-db17b036e580 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:51.220871393 +0000 UTC m=+40.055758745 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c6274f2-a3f1-4130-972b-db17b036e580-metrics-tls") pod "dns-default-nm7gc" (UID: "7c6274f2-a3f1-4130-972b-db17b036e580") : secret "dns-default-metrics-tls" not found
Jul 06 20:49:49.869763 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:49.869520    2559 generic.go:356] "Generic (PLEG): container finished" podID="a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2" containerID="764c964daf2b6bebc1dc51a841267fcba57d83efa4f7064ff6657cdc6d3d2ee5" exitCode=0
Jul 06 20:49:49.870174 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:49.869602    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wf6lk" event={"ID":"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2","Type":"ContainerDied","Data":"764c964daf2b6bebc1dc51a841267fcba57d83efa4f7064ff6657cdc6d3d2ee5"}
Jul 06 20:49:50.875785 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:50.875752    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wf6lk" event={"ID":"a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2","Type":"ContainerStarted","Data":"1f8ff14f05b5dcfe57ce5d63b188ece14b61a062a2020ee4be43f683ce1d2ecb"}
Jul 06 20:49:50.877126 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:50.877102    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dhdnl" event={"ID":"6524b812-b7e8-440c-891d-c219ad395486","Type":"ContainerStarted","Data":"59def74b63a7cf7a3804c494fd6e43295d17360383736c155840bac53b99c6cd"}
Jul 06 20:49:50.905479 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:50.905366    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wf6lk" podStartSLOduration=8.788033826 podStartE2EDuration="38.905343737s" totalImagesPullingTime="30.117309911s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:50.904206597 +0000 UTC m=+39.739093961" watchObservedRunningTime="2026-07-06 20:49:50.905343737 +0000 UTC m=+39.740231105"
Jul 06 20:49:51.235165 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:51.235076    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c6274f2-a3f1-4130-972b-db17b036e580-metrics-tls\") pod \"dns-default-nm7gc\" (UID: \"7c6274f2-a3f1-4130-972b-db17b036e580\") " pod="openshift-dns/dns-default-nm7gc"
Jul 06 20:49:51.235312 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:51.235218    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:49:51.235312 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:51.235282    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c6274f2-a3f1-4130-972b-db17b036e580-metrics-tls podName:7c6274f2-a3f1-4130-972b-db17b036e580 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:55.235266334 +0000 UTC m=+44.070153676 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c6274f2-a3f1-4130-972b-db17b036e580-metrics-tls") pod "dns-default-nm7gc" (UID: "7c6274f2-a3f1-4130-972b-db17b036e580") : secret "dns-default-metrics-tls" not found
Jul 06 20:49:55.262958 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:49:55.262920    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c6274f2-a3f1-4130-972b-db17b036e580-metrics-tls\") pod \"dns-default-nm7gc\" (UID: \"7c6274f2-a3f1-4130-972b-db17b036e580\") " pod="openshift-dns/dns-default-nm7gc"
Jul 06 20:49:55.263456 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:55.263068    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:49:55.263456 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:49:55.263131    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c6274f2-a3f1-4130-972b-db17b036e580-metrics-tls podName:7c6274f2-a3f1-4130-972b-db17b036e580 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:03.263115165 +0000 UTC m=+52.098002507 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c6274f2-a3f1-4130-972b-db17b036e580-metrics-tls") pod "dns-default-nm7gc" (UID: "7c6274f2-a3f1-4130-972b-db17b036e580") : secret "dns-default-metrics-tls" not found
Jul 06 20:50:03.318188 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:03.318151    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c6274f2-a3f1-4130-972b-db17b036e580-metrics-tls\") pod \"dns-default-nm7gc\" (UID: \"7c6274f2-a3f1-4130-972b-db17b036e580\") " pod="openshift-dns/dns-default-nm7gc"
Jul 06 20:50:03.318674 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:03.318330    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:50:03.318674 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:03.318393    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c6274f2-a3f1-4130-972b-db17b036e580-metrics-tls podName:7c6274f2-a3f1-4130-972b-db17b036e580 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:19.31837682 +0000 UTC m=+68.153264164 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c6274f2-a3f1-4130-972b-db17b036e580-metrics-tls") pod "dns-default-nm7gc" (UID: "7c6274f2-a3f1-4130-972b-db17b036e580") : secret "dns-default-metrics-tls" not found
Jul 06 20:50:04.806672 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.806624    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dhdnl" podStartSLOduration=15.568317001 podStartE2EDuration="17.806593142s" totalImagesPullingTime="2.238276141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:50.922645894 +0000 UTC m=+39.757533280" watchObservedRunningTime="2026-07-06 20:50:04.806593142 +0000 UTC m=+53.641480531"
Jul 06 20:50:04.807097 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.806899    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-h89pn"]
Jul 06 20:50:04.821517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.821493    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-h89pn"]
Jul 06 20:50:04.821643 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.821579    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-h89pn"
Jul 06 20:50:04.824104 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.824078    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 06 20:50:04.824213 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.824078    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 06 20:50:04.825117 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.825095    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mhwxz\""
Jul 06 20:50:04.825370 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.825355    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 06 20:50:04.825911 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.825885    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 06 20:50:04.851663 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.851640    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 06 20:50:04.912743 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.912717    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s9crg"]
Jul 06 20:50:04.928326 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.928299    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d43e04c1-782e-409e-a666-d1e2994fb8a5-snapshots\") pod \"insights-operator-5c847dcfc4-h89pn\" (UID: \"d43e04c1-782e-409e-a666-d1e2994fb8a5\") " pod="openshift-insights/insights-operator-5c847dcfc4-h89pn"
Jul 06 20:50:04.928425 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.928333    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d43e04c1-782e-409e-a666-d1e2994fb8a5-tmp\") pod \"insights-operator-5c847dcfc4-h89pn\" (UID: \"d43e04c1-782e-409e-a666-d1e2994fb8a5\") " pod="openshift-insights/insights-operator-5c847dcfc4-h89pn"
Jul 06 20:50:04.928425 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.928362    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d43e04c1-782e-409e-a666-d1e2994fb8a5-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-h89pn\" (UID: \"d43e04c1-782e-409e-a666-d1e2994fb8a5\") " pod="openshift-insights/insights-operator-5c847dcfc4-h89pn"
Jul 06 20:50:04.928530 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.928457    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d43e04c1-782e-409e-a666-d1e2994fb8a5-serving-cert\") pod \"insights-operator-5c847dcfc4-h89pn\" (UID: \"d43e04c1-782e-409e-a666-d1e2994fb8a5\") " pod="openshift-insights/insights-operator-5c847dcfc4-h89pn"
Jul 06 20:50:04.928530 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.928485    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc986\" (UniqueName: \"kubernetes.io/projected/d43e04c1-782e-409e-a666-d1e2994fb8a5-kube-api-access-gc986\") pod \"insights-operator-5c847dcfc4-h89pn\" (UID: \"d43e04c1-782e-409e-a666-d1e2994fb8a5\") " pod="openshift-insights/insights-operator-5c847dcfc4-h89pn"
Jul 06 20:50:04.928530 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.928506    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d43e04c1-782e-409e-a666-d1e2994fb8a5-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-h89pn\" (UID: \"d43e04c1-782e-409e-a666-d1e2994fb8a5\") " pod="openshift-insights/insights-operator-5c847dcfc4-h89pn"
Jul 06 20:50:04.936396 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.936378    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s9crg"
Jul 06 20:50:04.936396 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.936387    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s9crg"]
Jul 06 20:50:04.938804 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.938784    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 06 20:50:04.938941 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.938902    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dzh7d\""
Jul 06 20:50:04.938941 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.938930    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 06 20:50:04.939069 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.939051    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 06 20:50:04.939150 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:04.939135    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 06 20:50:05.029323 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.029294    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d43e04c1-782e-409e-a666-d1e2994fb8a5-snapshots\") pod \"insights-operator-5c847dcfc4-h89pn\" (UID: \"d43e04c1-782e-409e-a666-d1e2994fb8a5\") " pod="openshift-insights/insights-operator-5c847dcfc4-h89pn"
Jul 06 20:50:05.029323 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.029330    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d43e04c1-782e-409e-a666-d1e2994fb8a5-tmp\") pod \"insights-operator-5c847dcfc4-h89pn\" (UID: \"d43e04c1-782e-409e-a666-d1e2994fb8a5\") " pod="openshift-insights/insights-operator-5c847dcfc4-h89pn"
Jul 06 20:50:05.029556 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.029350    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqmk4\" (UniqueName: \"kubernetes.io/projected/324b6756-b968-4a97-acda-b44144b81452-kube-api-access-lqmk4\") pod \"cluster-monitoring-operator-64786d9cbb-s9crg\" (UID: \"324b6756-b968-4a97-acda-b44144b81452\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s9crg"
Jul 06 20:50:05.029556 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.029399    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d43e04c1-782e-409e-a666-d1e2994fb8a5-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-h89pn\" (UID: \"d43e04c1-782e-409e-a666-d1e2994fb8a5\") " pod="openshift-insights/insights-operator-5c847dcfc4-h89pn"
Jul 06 20:50:05.029556 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.029431    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/324b6756-b968-4a97-acda-b44144b81452-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-s9crg\" (UID: \"324b6756-b968-4a97-acda-b44144b81452\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s9crg"
Jul 06 20:50:05.029556 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.029509    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d43e04c1-782e-409e-a666-d1e2994fb8a5-serving-cert\") pod \"insights-operator-5c847dcfc4-h89pn\" (UID: \"d43e04c1-782e-409e-a666-d1e2994fb8a5\") " pod="openshift-insights/insights-operator-5c847dcfc4-h89pn"
Jul 06 20:50:05.029759 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.029649    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc986\" (UniqueName: \"kubernetes.io/projected/d43e04c1-782e-409e-a666-d1e2994fb8a5-kube-api-access-gc986\") pod \"insights-operator-5c847dcfc4-h89pn\" (UID: \"d43e04c1-782e-409e-a666-d1e2994fb8a5\") " pod="openshift-insights/insights-operator-5c847dcfc4-h89pn"
Jul 06 20:50:05.029759 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.029693    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/324b6756-b968-4a97-acda-b44144b81452-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-s9crg\" (UID: \"324b6756-b968-4a97-acda-b44144b81452\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s9crg"
Jul 06 20:50:05.029759 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.029725    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d43e04c1-782e-409e-a666-d1e2994fb8a5-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-h89pn\" (UID: \"d43e04c1-782e-409e-a666-d1e2994fb8a5\") " pod="openshift-insights/insights-operator-5c847dcfc4-h89pn"
Jul 06 20:50:05.029974 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.029956    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d43e04c1-782e-409e-a666-d1e2994fb8a5-tmp\") pod \"insights-operator-5c847dcfc4-h89pn\" (UID: \"d43e04c1-782e-409e-a666-d1e2994fb8a5\") " pod="openshift-insights/insights-operator-5c847dcfc4-h89pn"
Jul 06 20:50:05.030047 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.030028    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d43e04c1-782e-409e-a666-d1e2994fb8a5-snapshots\") pod \"insights-operator-5c847dcfc4-h89pn\" (UID: \"d43e04c1-782e-409e-a666-d1e2994fb8a5\") " pod="openshift-insights/insights-operator-5c847dcfc4-h89pn"
Jul 06 20:50:05.030270 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.030250    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d43e04c1-782e-409e-a666-d1e2994fb8a5-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-h89pn\" (UID: \"d43e04c1-782e-409e-a666-d1e2994fb8a5\") " pod="openshift-insights/insights-operator-5c847dcfc4-h89pn"
Jul 06 20:50:05.030365 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.030343    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d43e04c1-782e-409e-a666-d1e2994fb8a5-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-h89pn\" (UID: \"d43e04c1-782e-409e-a666-d1e2994fb8a5\") " pod="openshift-insights/insights-operator-5c847dcfc4-h89pn"
Jul 06 20:50:05.033025 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.033008    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d43e04c1-782e-409e-a666-d1e2994fb8a5-serving-cert\") pod \"insights-operator-5c847dcfc4-h89pn\" (UID: \"d43e04c1-782e-409e-a666-d1e2994fb8a5\") " pod="openshift-insights/insights-operator-5c847dcfc4-h89pn"
Jul 06 20:50:05.039529 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.039510    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc986\" (UniqueName: \"kubernetes.io/projected/d43e04c1-782e-409e-a666-d1e2994fb8a5-kube-api-access-gc986\") pod \"insights-operator-5c847dcfc4-h89pn\" (UID: \"d43e04c1-782e-409e-a666-d1e2994fb8a5\") " pod="openshift-insights/insights-operator-5c847dcfc4-h89pn"
Jul 06 20:50:05.130810 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.130714    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-h89pn"
Jul 06 20:50:05.130962 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.130864    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqmk4\" (UniqueName: \"kubernetes.io/projected/324b6756-b968-4a97-acda-b44144b81452-kube-api-access-lqmk4\") pod \"cluster-monitoring-operator-64786d9cbb-s9crg\" (UID: \"324b6756-b968-4a97-acda-b44144b81452\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s9crg"
Jul 06 20:50:05.130962 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.130914    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/324b6756-b968-4a97-acda-b44144b81452-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-s9crg\" (UID: \"324b6756-b968-4a97-acda-b44144b81452\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s9crg"
Jul 06 20:50:05.131029 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.130966    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/324b6756-b968-4a97-acda-b44144b81452-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-s9crg\" (UID: \"324b6756-b968-4a97-acda-b44144b81452\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s9crg"
Jul 06 20:50:05.131103 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:05.131085    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 06 20:50:05.131173 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:05.131163    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/324b6756-b968-4a97-acda-b44144b81452-cluster-monitoring-operator-tls podName:324b6756-b968-4a97-acda-b44144b81452 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:05.631145972 +0000 UTC m=+54.466033333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/324b6756-b968-4a97-acda-b44144b81452-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-s9crg" (UID: "324b6756-b968-4a97-acda-b44144b81452") : secret "cluster-monitoring-operator-tls" not found
Jul 06 20:50:05.131605 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.131586    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/324b6756-b968-4a97-acda-b44144b81452-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-s9crg\" (UID: \"324b6756-b968-4a97-acda-b44144b81452\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s9crg"
Jul 06 20:50:05.141409 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.141380    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqmk4\" (UniqueName: \"kubernetes.io/projected/324b6756-b968-4a97-acda-b44144b81452-kube-api-access-lqmk4\") pod \"cluster-monitoring-operator-64786d9cbb-s9crg\" (UID: \"324b6756-b968-4a97-acda-b44144b81452\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s9crg"
Jul 06 20:50:05.310728 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.310685    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-h89pn"]
Jul 06 20:50:05.313793 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:50:05.313766    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43e04c1_782e_409e_a666_d1e2994fb8a5.slice/crio-0c9712c4c4fa9a2b80529fd09b87167050310adef116b85c9f32e67bb78438b1 WatchSource:0}: Error finding container 0c9712c4c4fa9a2b80529fd09b87167050310adef116b85c9f32e67bb78438b1: Status 404 returned error can't find the container with id 0c9712c4c4fa9a2b80529fd09b87167050310adef116b85c9f32e67bb78438b1
Jul 06 20:50:05.635323 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.635291    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/324b6756-b968-4a97-acda-b44144b81452-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-s9crg\" (UID: \"324b6756-b968-4a97-acda-b44144b81452\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s9crg"
Jul 06 20:50:05.635480 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:05.635453    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 06 20:50:05.635533 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:05.635522    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/324b6756-b968-4a97-acda-b44144b81452-cluster-monitoring-operator-tls podName:324b6756-b968-4a97-acda-b44144b81452 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:06.635506679 +0000 UTC m=+55.470394026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/324b6756-b968-4a97-acda-b44144b81452-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-s9crg" (UID: "324b6756-b968-4a97-acda-b44144b81452") : secret "cluster-monitoring-operator-tls" not found
Jul 06 20:50:05.915471 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:05.915393    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-h89pn" event={"ID":"d43e04c1-782e-409e-a666-d1e2994fb8a5","Type":"ContainerStarted","Data":"0c9712c4c4fa9a2b80529fd09b87167050310adef116b85c9f32e67bb78438b1"}
Jul 06 20:50:06.643827 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:06.643792    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/324b6756-b968-4a97-acda-b44144b81452-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-s9crg\" (UID: \"324b6756-b968-4a97-acda-b44144b81452\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s9crg"
Jul 06 20:50:06.643999 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:06.643981    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 06 20:50:06.644075 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:06.644063    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/324b6756-b968-4a97-acda-b44144b81452-cluster-monitoring-operator-tls podName:324b6756-b968-4a97-acda-b44144b81452 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:08.644042225 +0000 UTC m=+57.478929570 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/324b6756-b968-4a97-acda-b44144b81452-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-s9crg" (UID: "324b6756-b968-4a97-acda-b44144b81452") : secret "cluster-monitoring-operator-tls" not found
Jul 06 20:50:07.922224 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:07.922187    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-h89pn" event={"ID":"d43e04c1-782e-409e-a666-d1e2994fb8a5","Type":"ContainerStarted","Data":"af37d7ffee4c3025d079e7b8ec68127822e1faa3b130b0cddaf823dc02ac940c"}
Jul 06 20:50:07.940398 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:07.940225    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-h89pn" podStartSLOduration=2.353660653 podStartE2EDuration="3.94020696s" totalImagesPullingTime="1.586546307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:07.93954502 +0000 UTC m=+56.774432397" watchObservedRunningTime="2026-07-06 20:50:07.94020696 +0000 UTC m=+56.775094316"
Jul 06 20:50:08.087694 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:08.087664    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gfdzd"]
Jul 06 20:50:08.089491 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:08.089476    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:08.091591 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:08.091568    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 06 20:50:08.091737 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:08.091587    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mz8wb\""
Jul 06 20:50:08.091787 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:08.091752    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 06 20:50:08.104068 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:08.104009    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gfdzd"]
Jul 06 20:50:08.257000 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:08.256917    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tbp4\" (UniqueName: \"kubernetes.io/projected/cc937b92-57db-4b45-9b3b-ad57d91ca45d-kube-api-access-2tbp4\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:08.257000 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:08.256984    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc937b92-57db-4b45-9b3b-ad57d91ca45d-crio-socket\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:08.257188 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:08.257013    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc937b92-57db-4b45-9b3b-ad57d91ca45d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:08.257188 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:08.257062    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc937b92-57db-4b45-9b3b-ad57d91ca45d-data-volume\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:08.257188 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:08.257147    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc937b92-57db-4b45-9b3b-ad57d91ca45d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:08.357788 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:08.357747    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc937b92-57db-4b45-9b3b-ad57d91ca45d-crio-socket\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:08.357788 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:08.357788    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc937b92-57db-4b45-9b3b-ad57d91ca45d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:08.358010 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:08.357824    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc937b92-57db-4b45-9b3b-ad57d91ca45d-data-volume\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:08.358010 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:08.357865    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc937b92-57db-4b45-9b3b-ad57d91ca45d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:08.358010 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:08.357888    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tbp4\" (UniqueName: \"kubernetes.io/projected/cc937b92-57db-4b45-9b3b-ad57d91ca45d-kube-api-access-2tbp4\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:08.358010 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:08.357889    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:50:08.358010 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:08.357950    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc937b92-57db-4b45-9b3b-ad57d91ca45d-crio-socket\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:08.358010 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:08.357981    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc937b92-57db-4b45-9b3b-ad57d91ca45d-kube-rbac-proxy-cm podName:cc937b92-57db-4b45-9b3b-ad57d91ca45d nodeName:}" failed. No retries permitted until 2026-07-06 20:50:08.857959732 +0000 UTC m=+57.692847099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc937b92-57db-4b45-9b3b-ad57d91ca45d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gfdzd" (UID: "cc937b92-57db-4b45-9b3b-ad57d91ca45d") : configmap "kube-rbac-proxy" not found
Jul 06 20:50:08.358010 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:08.358001    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:50:08.358327 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:08.358038    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc937b92-57db-4b45-9b3b-ad57d91ca45d-insights-runtime-extractor-tls podName:cc937b92-57db-4b45-9b3b-ad57d91ca45d nodeName:}" failed. No retries permitted until 2026-07-06 20:50:08.858027219 +0000 UTC m=+57.692914577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc937b92-57db-4b45-9b3b-ad57d91ca45d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gfdzd" (UID: "cc937b92-57db-4b45-9b3b-ad57d91ca45d") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:50:08.358327 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:08.358225    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc937b92-57db-4b45-9b3b-ad57d91ca45d-data-volume\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:08.368805 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:08.368778    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tbp4\" (UniqueName: \"kubernetes.io/projected/cc937b92-57db-4b45-9b3b-ad57d91ca45d-kube-api-access-2tbp4\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:08.660161 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:08.660121    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/324b6756-b968-4a97-acda-b44144b81452-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-s9crg\" (UID: \"324b6756-b968-4a97-acda-b44144b81452\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s9crg"
Jul 06 20:50:08.660332 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:08.660264    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 06 20:50:08.660332 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:08.660327    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/324b6756-b968-4a97-acda-b44144b81452-cluster-monitoring-operator-tls podName:324b6756-b968-4a97-acda-b44144b81452 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:12.660313262 +0000 UTC m=+61.495200606 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/324b6756-b968-4a97-acda-b44144b81452-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-s9crg" (UID: "324b6756-b968-4a97-acda-b44144b81452") : secret "cluster-monitoring-operator-tls" not found
Jul 06 20:50:08.861460 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:08.861422    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc937b92-57db-4b45-9b3b-ad57d91ca45d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:08.861641 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:08.861498    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc937b92-57db-4b45-9b3b-ad57d91ca45d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:08.861641 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:08.861596    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:50:08.861722 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:08.861689    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc937b92-57db-4b45-9b3b-ad57d91ca45d-insights-runtime-extractor-tls podName:cc937b92-57db-4b45-9b3b-ad57d91ca45d nodeName:}" failed. No retries permitted until 2026-07-06 20:50:09.861674123 +0000 UTC m=+58.696561468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc937b92-57db-4b45-9b3b-ad57d91ca45d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gfdzd" (UID: "cc937b92-57db-4b45-9b3b-ad57d91ca45d") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:50:08.861722 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:08.861596    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:50:08.861802 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:08.861767    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc937b92-57db-4b45-9b3b-ad57d91ca45d-kube-rbac-proxy-cm podName:cc937b92-57db-4b45-9b3b-ad57d91ca45d nodeName:}" failed. No retries permitted until 2026-07-06 20:50:09.861754859 +0000 UTC m=+58.696642204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc937b92-57db-4b45-9b3b-ad57d91ca45d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gfdzd" (UID: "cc937b92-57db-4b45-9b3b-ad57d91ca45d") : configmap "kube-rbac-proxy" not found
Jul 06 20:50:09.853643 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:09.853604    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mzmqn_036a3e17-12a4-418f-b550-752a6ff9694e/dns-node-resolver/0.log"
Jul 06 20:50:09.870102 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:09.870072    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc937b92-57db-4b45-9b3b-ad57d91ca45d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:09.870221 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:09.870141    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc937b92-57db-4b45-9b3b-ad57d91ca45d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:09.870221 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:09.870193    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:50:09.870292 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:09.870225    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:50:09.870292 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:09.870255    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc937b92-57db-4b45-9b3b-ad57d91ca45d-kube-rbac-proxy-cm podName:cc937b92-57db-4b45-9b3b-ad57d91ca45d nodeName:}" failed. No retries permitted until 2026-07-06 20:50:11.870234872 +0000 UTC m=+60.705122215 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc937b92-57db-4b45-9b3b-ad57d91ca45d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gfdzd" (UID: "cc937b92-57db-4b45-9b3b-ad57d91ca45d") : configmap "kube-rbac-proxy" not found
Jul 06 20:50:09.870292 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:09.870272    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc937b92-57db-4b45-9b3b-ad57d91ca45d-insights-runtime-extractor-tls podName:cc937b92-57db-4b45-9b3b-ad57d91ca45d nodeName:}" failed. No retries permitted until 2026-07-06 20:50:11.870261758 +0000 UTC m=+60.705149102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc937b92-57db-4b45-9b3b-ad57d91ca45d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gfdzd" (UID: "cc937b92-57db-4b45-9b3b-ad57d91ca45d") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:50:11.053347 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:11.053319    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rbr6h_5d4b9056-4c7c-43fd-a2b6-6d617f5cf503/node-ca/0.log"
Jul 06 20:50:11.454047 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:11.453945    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dhdnl_6524b812-b7e8-440c-891d-c219ad395486/serve-healthcheck-canary/0.log"
Jul 06 20:50:11.886659 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:11.886626    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc937b92-57db-4b45-9b3b-ad57d91ca45d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:11.886821 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:11.886705    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc937b92-57db-4b45-9b3b-ad57d91ca45d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:11.886821 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:11.886748    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:50:11.886821 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:11.886812    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc937b92-57db-4b45-9b3b-ad57d91ca45d-kube-rbac-proxy-cm podName:cc937b92-57db-4b45-9b3b-ad57d91ca45d nodeName:}" failed. No retries permitted until 2026-07-06 20:50:15.886795998 +0000 UTC m=+64.721683342 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc937b92-57db-4b45-9b3b-ad57d91ca45d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gfdzd" (UID: "cc937b92-57db-4b45-9b3b-ad57d91ca45d") : configmap "kube-rbac-proxy" not found
Jul 06 20:50:11.886821 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:11.886818    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:50:11.886958 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:11.886862    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc937b92-57db-4b45-9b3b-ad57d91ca45d-insights-runtime-extractor-tls podName:cc937b92-57db-4b45-9b3b-ad57d91ca45d nodeName:}" failed. No retries permitted until 2026-07-06 20:50:15.886847685 +0000 UTC m=+64.721735028 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc937b92-57db-4b45-9b3b-ad57d91ca45d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gfdzd" (UID: "cc937b92-57db-4b45-9b3b-ad57d91ca45d") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:50:12.692884 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:12.692848    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/324b6756-b968-4a97-acda-b44144b81452-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-s9crg\" (UID: \"324b6756-b968-4a97-acda-b44144b81452\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s9crg"
Jul 06 20:50:12.693258 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:12.692998    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 06 20:50:12.693258 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:12.693066    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/324b6756-b968-4a97-acda-b44144b81452-cluster-monitoring-operator-tls podName:324b6756-b968-4a97-acda-b44144b81452 nodeName:}" failed. No retries permitted until 2026-07-06 20:50:20.693051986 +0000 UTC m=+69.527939329 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/324b6756-b968-4a97-acda-b44144b81452-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-s9crg" (UID: "324b6756-b968-4a97-acda-b44144b81452") : secret "cluster-monitoring-operator-tls" not found
Jul 06 20:50:12.854975 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:12.854947    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pt6t9"
Jul 06 20:50:15.918518 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:15.918483    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc937b92-57db-4b45-9b3b-ad57d91ca45d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:15.919023 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:15.918574    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc937b92-57db-4b45-9b3b-ad57d91ca45d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:15.919023 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:15.918604    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:50:15.919023 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:15.918678    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc937b92-57db-4b45-9b3b-ad57d91ca45d-kube-rbac-proxy-cm podName:cc937b92-57db-4b45-9b3b-ad57d91ca45d nodeName:}" failed. No retries permitted until 2026-07-06 20:50:23.91866393 +0000 UTC m=+72.753551274 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc937b92-57db-4b45-9b3b-ad57d91ca45d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gfdzd" (UID: "cc937b92-57db-4b45-9b3b-ad57d91ca45d") : configmap "kube-rbac-proxy" not found
Jul 06 20:50:15.919023 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:15.918686    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:50:15.919023 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:15.918725    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc937b92-57db-4b45-9b3b-ad57d91ca45d-insights-runtime-extractor-tls podName:cc937b92-57db-4b45-9b3b-ad57d91ca45d nodeName:}" failed. No retries permitted until 2026-07-06 20:50:23.918714359 +0000 UTC m=+72.753601701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc937b92-57db-4b45-9b3b-ad57d91ca45d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gfdzd" (UID: "cc937b92-57db-4b45-9b3b-ad57d91ca45d") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:50:17.328769 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:17.328738    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs\") pod \"network-metrics-daemon-ckj9h\" (UID: \"ad24b800-9783-47dc-aaa1-4d2bbc01afb8\") " pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:50:17.331123 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:17.331104    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 20:50:17.339148 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:17.339127    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 06 20:50:17.339233 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:17.339190    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs podName:ad24b800-9783-47dc-aaa1-4d2bbc01afb8 nodeName:}" failed. No retries permitted until 2026-07-06 20:51:21.339171147 +0000 UTC m=+130.174058497 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs") pod "network-metrics-daemon-ckj9h" (UID: "ad24b800-9783-47dc-aaa1-4d2bbc01afb8") : secret "metrics-daemon-secret" not found
Jul 06 20:50:17.429973 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:17.429947    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret\") pod \"global-pull-secret-syncer-khnhx\" (UID: \"c260a02b-d5f8-48c9-894b-44c6310ef087\") " pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:50:17.430114 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:17.429988    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkzpl\" (UniqueName: \"kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl\") pod \"network-check-target-p9bcs\" (UID: \"806f1f1d-5cea-4e14-b4e8-a293bd4b3393\") " pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:50:17.432474 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:17.432454    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 20:50:17.432564 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:17.432482    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 20:50:17.442455 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:17.442429    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 20:50:17.442533 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:17.442473    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c260a02b-d5f8-48c9-894b-44c6310ef087-original-pull-secret\") pod \"global-pull-secret-syncer-khnhx\" (UID: \"c260a02b-d5f8-48c9-894b-44c6310ef087\") " pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:50:17.453669 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:17.453652    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkzpl\" (UniqueName: \"kubernetes.io/projected/806f1f1d-5cea-4e14-b4e8-a293bd4b3393-kube-api-access-fkzpl\") pod \"network-check-target-p9bcs\" (UID: \"806f1f1d-5cea-4e14-b4e8-a293bd4b3393\") " pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:50:17.466697 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:17.466682    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hgfwv\""
Jul 06 20:50:17.475648 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:17.475632    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:50:17.595538 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:17.595467    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p9bcs"]
Jul 06 20:50:17.598301 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:50:17.598278    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806f1f1d_5cea_4e14_b4e8_a293bd4b3393.slice/crio-18d92e859412a7eda490af302cf490484e511e3522ffd5204214bb096f65f91d WatchSource:0}: Error finding container 18d92e859412a7eda490af302cf490484e511e3522ffd5204214bb096f65f91d: Status 404 returned error can't find the container with id 18d92e859412a7eda490af302cf490484e511e3522ffd5204214bb096f65f91d
Jul 06 20:50:17.662516 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:17.662486    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-khnhx"
Jul 06 20:50:17.799831 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:17.799798    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-khnhx"]
Jul 06 20:50:17.803808 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:50:17.803780    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc260a02b_d5f8_48c9_894b_44c6310ef087.slice/crio-814f5db510eafdb68166475f0bc696f2ed09b2c0dd324a050b118a06fbbd5966 WatchSource:0}: Error finding container 814f5db510eafdb68166475f0bc696f2ed09b2c0dd324a050b118a06fbbd5966: Status 404 returned error can't find the container with id 814f5db510eafdb68166475f0bc696f2ed09b2c0dd324a050b118a06fbbd5966
Jul 06 20:50:17.946491 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:17.946411    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-khnhx" event={"ID":"c260a02b-d5f8-48c9-894b-44c6310ef087","Type":"ContainerStarted","Data":"814f5db510eafdb68166475f0bc696f2ed09b2c0dd324a050b118a06fbbd5966"}
Jul 06 20:50:17.947486 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:17.947456    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p9bcs" event={"ID":"806f1f1d-5cea-4e14-b4e8-a293bd4b3393","Type":"ContainerStarted","Data":"18d92e859412a7eda490af302cf490484e511e3522ffd5204214bb096f65f91d"}
Jul 06 20:50:19.344003 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:19.343819    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c6274f2-a3f1-4130-972b-db17b036e580-metrics-tls\") pod \"dns-default-nm7gc\" (UID: \"7c6274f2-a3f1-4130-972b-db17b036e580\") " pod="openshift-dns/dns-default-nm7gc"
Jul 06 20:50:19.346819 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:19.346794    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c6274f2-a3f1-4130-972b-db17b036e580-metrics-tls\") pod \"dns-default-nm7gc\" (UID: \"7c6274f2-a3f1-4130-972b-db17b036e580\") " pod="openshift-dns/dns-default-nm7gc"
Jul 06 20:50:19.356939 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:19.356916    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cpffn\""
Jul 06 20:50:19.364943 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:19.364909    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nm7gc"
Jul 06 20:50:19.532785 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:19.532742    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nm7gc"]
Jul 06 20:50:19.535721 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:50:19.535690    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6274f2_a3f1_4130_972b_db17b036e580.slice/crio-113822bc79c1e7e19ca6b960023f5f4ab7a8c511b79bb10159493c08e8e8de3d WatchSource:0}: Error finding container 113822bc79c1e7e19ca6b960023f5f4ab7a8c511b79bb10159493c08e8e8de3d: Status 404 returned error can't find the container with id 113822bc79c1e7e19ca6b960023f5f4ab7a8c511b79bb10159493c08e8e8de3d
Jul 06 20:50:19.955580 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:19.955533    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nm7gc" event={"ID":"7c6274f2-a3f1-4130-972b-db17b036e580","Type":"ContainerStarted","Data":"113822bc79c1e7e19ca6b960023f5f4ab7a8c511b79bb10159493c08e8e8de3d"}
Jul 06 20:50:20.753988 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:20.753951    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/324b6756-b968-4a97-acda-b44144b81452-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-s9crg\" (UID: \"324b6756-b968-4a97-acda-b44144b81452\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s9crg"
Jul 06 20:50:20.756393 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:20.756367    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/324b6756-b968-4a97-acda-b44144b81452-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-s9crg\" (UID: \"324b6756-b968-4a97-acda-b44144b81452\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s9crg"
Jul 06 20:50:20.847300 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:20.847269    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dzh7d\""
Jul 06 20:50:20.855065 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:20.855041    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s9crg"
Jul 06 20:50:22.767520 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:22.767458    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s9crg"]
Jul 06 20:50:22.772981 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:50:22.772948    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324b6756_b968_4a97_acda_b44144b81452.slice/crio-e534b7657e05f60bb51d1bc351712006bf126c7694ab651b80647879396ea45f WatchSource:0}: Error finding container e534b7657e05f60bb51d1bc351712006bf126c7694ab651b80647879396ea45f: Status 404 returned error can't find the container with id e534b7657e05f60bb51d1bc351712006bf126c7694ab651b80647879396ea45f
Jul 06 20:50:22.964918 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:22.964833    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-khnhx" event={"ID":"c260a02b-d5f8-48c9-894b-44c6310ef087","Type":"ContainerStarted","Data":"a0c5fc511cdc9c30eb58066a3dc19082a75c825b7e1e4f076a6f73a517f89bef"}
Jul 06 20:50:22.966622 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:22.966584    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nm7gc" event={"ID":"7c6274f2-a3f1-4130-972b-db17b036e580","Type":"ContainerStarted","Data":"66a49beeda9bad1241a8049b283ccc94c7eada6e218c077ff1b71ddeed1e2bfb"}
Jul 06 20:50:22.966759 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:22.966643    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nm7gc" event={"ID":"7c6274f2-a3f1-4130-972b-db17b036e580","Type":"ContainerStarted","Data":"c36da06e0a2e51a1107e366da2625c921225e4314941f5d7f4fae57fb6145a45"}
Jul 06 20:50:22.966759 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:22.966681    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nm7gc"
Jul 06 20:50:22.971249 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:22.968431    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p9bcs" event={"ID":"806f1f1d-5cea-4e14-b4e8-a293bd4b3393","Type":"ContainerStarted","Data":"0735235c2a6eab632e3ca27301017297c086a7d0679ae39a1fe85db12ac9736f"}
Jul 06 20:50:22.971249 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:22.968834    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:50:22.972363 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:22.972343    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s9crg" event={"ID":"324b6756-b968-4a97-acda-b44144b81452","Type":"ContainerStarted","Data":"e534b7657e05f60bb51d1bc351712006bf126c7694ab651b80647879396ea45f"}
Jul 06 20:50:22.984129 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:22.984087    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-khnhx" podStartSLOduration=65.150467239 podStartE2EDuration="1m9.984074245s" totalImagesPullingTime="4.833607006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:22.983468782 +0000 UTC m=+71.818356147" watchObservedRunningTime="2026-07-06 20:50:22.984074245 +0000 UTC m=+71.818961612"
Jul 06 20:50:23.002503 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:23.002446    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nm7gc" podStartSLOduration=32.895054382 podStartE2EDuration="36.002434139s" totalImagesPullingTime="3.107379757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:23.001532399 +0000 UTC m=+71.836419790" watchObservedRunningTime="2026-07-06 20:50:23.002434139 +0000 UTC m=+71.837321512"
Jul 06 20:50:23.024517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:23.024466    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-p9bcs" podStartSLOduration=66.00468842 podStartE2EDuration="1m11.024451808s" totalImagesPullingTime="5.019763388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:23.02428412 +0000 UTC m=+71.859171506" watchObservedRunningTime="2026-07-06 20:50:23.024451808 +0000 UTC m=+71.859339174"
Jul 06 20:50:23.979828 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:23.979794    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc937b92-57db-4b45-9b3b-ad57d91ca45d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:23.980224 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:23.979876    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc937b92-57db-4b45-9b3b-ad57d91ca45d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:23.980311 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:23.980291    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc937b92-57db-4b45-9b3b-ad57d91ca45d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:23.982199 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:23.982179    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc937b92-57db-4b45-9b3b-ad57d91ca45d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gfdzd\" (UID: \"cc937b92-57db-4b45-9b3b-ad57d91ca45d\") " pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:24.000293 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:24.000268    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mz8wb\""
Jul 06 20:50:24.008273 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:24.008252    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gfdzd"
Jul 06 20:50:24.461203 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:24.461096    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gfdzd"]
Jul 06 20:50:24.465408 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:50:24.465366    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc937b92_57db_4b45_9b3b_ad57d91ca45d.slice/crio-22fa6cea9ba3a54c07fa42e9d2d95e63bf3991123b3c87fb97041e0bb1639bab WatchSource:0}: Error finding container 22fa6cea9ba3a54c07fa42e9d2d95e63bf3991123b3c87fb97041e0bb1639bab: Status 404 returned error can't find the container with id 22fa6cea9ba3a54c07fa42e9d2d95e63bf3991123b3c87fb97041e0bb1639bab
Jul 06 20:50:24.983433 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:24.983389    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s9crg" event={"ID":"324b6756-b968-4a97-acda-b44144b81452","Type":"ContainerStarted","Data":"7b8dbda06d13f27fce4f258032167859134d4086bc4f26d4dd63e265aff55324"}
Jul 06 20:50:24.987667 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:24.987636    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gfdzd" event={"ID":"cc937b92-57db-4b45-9b3b-ad57d91ca45d","Type":"ContainerStarted","Data":"22fa6cea9ba3a54c07fa42e9d2d95e63bf3991123b3c87fb97041e0bb1639bab"}
Jul 06 20:50:25.004137 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:25.003534    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s9crg" podStartSLOduration=19.419505728 podStartE2EDuration="21.00351444s" totalImagesPullingTime="1.584008712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:25.00222249 +0000 UTC m=+73.837109854" watchObservedRunningTime="2026-07-06 20:50:25.00351444 +0000 UTC m=+73.838401808"
Jul 06 20:50:29.002848 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:29.002814    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gfdzd" event={"ID":"cc937b92-57db-4b45-9b3b-ad57d91ca45d","Type":"ContainerStarted","Data":"f831079552c37258abfcdf5bca6931c7c3ce9ee814515a3c69bcec64c9ae1265"}
Jul 06 20:50:29.003232 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:29.002875    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gfdzd" event={"ID":"cc937b92-57db-4b45-9b3b-ad57d91ca45d","Type":"ContainerStarted","Data":"add7749f6373652235a6079630a522a87b33f07b9e8cf3838aa6c085e68f1019"}
Jul 06 20:50:32.015230 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.015190    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gfdzd" event={"ID":"cc937b92-57db-4b45-9b3b-ad57d91ca45d","Type":"ContainerStarted","Data":"127503072bff068ed2bf23bbe1a323c6622ea7715e420ab1d3f7d91faf9136f3"}
Jul 06 20:50:32.038405 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.038363    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gfdzd" podStartSLOduration=17.30563998 podStartE2EDuration="24.038348256s" totalImagesPullingTime="6.732708276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:50:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:32.037475124 +0000 UTC m=+80.872362499" watchObservedRunningTime="2026-07-06 20:50:32.038348256 +0000 UTC m=+80.873235633"
Jul 06 20:50:32.387494 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.387454    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b996x"]
Jul 06 20:50:32.390111 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.390090    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cb76c4bf-rkz24"]
Jul 06 20:50:32.390233 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.390219    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b996x"
Jul 06 20:50:32.391909 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.391892    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.392774 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.392755    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-spzlt\""
Jul 06 20:50:32.393088 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.393075    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 06 20:50:32.393574 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.393557    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 06 20:50:32.393675 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.393577    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:50:32.398848 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.398826    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 06 20:50:32.398986 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.398969    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 06 20:50:32.400835 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.400818    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 06 20:50:32.401106 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.401093    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ldn5x\""
Jul 06 20:50:32.406836 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.406819    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 06 20:50:32.411569 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.411550    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b996x"]
Jul 06 20:50:32.415903 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.415884    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-78d6fff86-dqqf7"]
Jul 06 20:50:32.417956 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.417942    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78d6fff86-dqqf7"
Jul 06 20:50:32.422156 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.422137    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 06 20:50:32.422779 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.422761    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 06 20:50:32.422913 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.422898    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 06 20:50:32.426197 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.426177    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 06 20:50:32.434300 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.434280    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cb76c4bf-rkz24"]
Jul 06 20:50:32.440408 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.440390    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-78d6fff86-dqqf7"]
Jul 06 20:50:32.443952 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.443931    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/feef5889-46b1-4518-af04-2a11d9fff0c8-installation-pull-secrets\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.444039 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.443962    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/feef5889-46b1-4518-af04-2a11d9fff0c8-registry-tls\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.444039 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.443988    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/feef5889-46b1-4518-af04-2a11d9fff0c8-image-registry-private-configuration\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.444039 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.444006    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/af0e795f-ce9d-4463-b889-b384a89262f3-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-b996x\" (UID: \"af0e795f-ce9d-4463-b889-b384a89262f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b996x"
Jul 06 20:50:32.444039 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.444033    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/feef5889-46b1-4518-af04-2a11d9fff0c8-ca-trust-extracted\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.444205 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.444057    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9h8j\" (UniqueName: \"kubernetes.io/projected/af0e795f-ce9d-4463-b889-b384a89262f3-kube-api-access-r9h8j\") pod \"cluster-samples-operator-56fcf86bf-b996x\" (UID: \"af0e795f-ce9d-4463-b889-b384a89262f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b996x"
Jul 06 20:50:32.444205 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.444075    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/feef5889-46b1-4518-af04-2a11d9fff0c8-bound-sa-token\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.444205 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.444092    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/feef5889-46b1-4518-af04-2a11d9fff0c8-registry-certificates\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.444205 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.444116    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdmjl\" (UniqueName: \"kubernetes.io/projected/feef5889-46b1-4518-af04-2a11d9fff0c8-kube-api-access-tdmjl\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.444205 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.444160    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/feef5889-46b1-4518-af04-2a11d9fff0c8-trusted-ca\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.545273 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.545242    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/feef5889-46b1-4518-af04-2a11d9fff0c8-installation-pull-secrets\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.545460 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.545282    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/feef5889-46b1-4518-af04-2a11d9fff0c8-registry-tls\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.545460 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.545305    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/feef5889-46b1-4518-af04-2a11d9fff0c8-image-registry-private-configuration\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.545460 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.545320    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/af0e795f-ce9d-4463-b889-b384a89262f3-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-b996x\" (UID: \"af0e795f-ce9d-4463-b889-b384a89262f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b996x"
Jul 06 20:50:32.545460 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.545346    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm5cg\" (UniqueName: \"kubernetes.io/projected/ff5890ec-30ec-4004-9877-5fde2033f4d6-kube-api-access-mm5cg\") pod \"klusterlet-addon-workmgr-78d6fff86-dqqf7\" (UID: \"ff5890ec-30ec-4004-9877-5fde2033f4d6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78d6fff86-dqqf7"
Jul 06 20:50:32.545460 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.545370    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/feef5889-46b1-4518-af04-2a11d9fff0c8-ca-trust-extracted\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.545460 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.545404    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9h8j\" (UniqueName: \"kubernetes.io/projected/af0e795f-ce9d-4463-b889-b384a89262f3-kube-api-access-r9h8j\") pod \"cluster-samples-operator-56fcf86bf-b996x\" (UID: \"af0e795f-ce9d-4463-b889-b384a89262f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b996x"
Jul 06 20:50:32.545460 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.545438    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/feef5889-46b1-4518-af04-2a11d9fff0c8-bound-sa-token\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.545815 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.545468    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/feef5889-46b1-4518-af04-2a11d9fff0c8-registry-certificates\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.545815 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.545494    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdmjl\" (UniqueName: \"kubernetes.io/projected/feef5889-46b1-4518-af04-2a11d9fff0c8-kube-api-access-tdmjl\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.545815 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.545522    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff5890ec-30ec-4004-9877-5fde2033f4d6-tmp\") pod \"klusterlet-addon-workmgr-78d6fff86-dqqf7\" (UID: \"ff5890ec-30ec-4004-9877-5fde2033f4d6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78d6fff86-dqqf7"
Jul 06 20:50:32.545815 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.545568    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/feef5889-46b1-4518-af04-2a11d9fff0c8-trusted-ca\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.545815 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.545590    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ff5890ec-30ec-4004-9877-5fde2033f4d6-klusterlet-config\") pod \"klusterlet-addon-workmgr-78d6fff86-dqqf7\" (UID: \"ff5890ec-30ec-4004-9877-5fde2033f4d6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78d6fff86-dqqf7"
Jul 06 20:50:32.546074 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.545883    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/feef5889-46b1-4518-af04-2a11d9fff0c8-ca-trust-extracted\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.546630 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.546592    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/feef5889-46b1-4518-af04-2a11d9fff0c8-registry-certificates\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.546735 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.546714    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/feef5889-46b1-4518-af04-2a11d9fff0c8-trusted-ca\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.548113 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.548083    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/feef5889-46b1-4518-af04-2a11d9fff0c8-installation-pull-secrets\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.548212 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.548083    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/feef5889-46b1-4518-af04-2a11d9fff0c8-image-registry-private-configuration\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.548361 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.548344    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/af0e795f-ce9d-4463-b889-b384a89262f3-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-b996x\" (UID: \"af0e795f-ce9d-4463-b889-b384a89262f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b996x"
Jul 06 20:50:32.548415 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.548389    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/feef5889-46b1-4518-af04-2a11d9fff0c8-registry-tls\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.551666 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.551642    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-n7nvx"]
Jul 06 20:50:32.554620 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.554547    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-8824w"]
Jul 06 20:50:32.554957 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.554813    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-n7nvx"
Jul 06 20:50:32.556691 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.556672    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-cxbrh"]
Jul 06 20:50:32.556813 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.556790    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-8824w"
Jul 06 20:50:32.558665 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.558648    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-6btsj"]
Jul 06 20:50:32.558786 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.558771    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-cxbrh"
Jul 06 20:50:32.560461 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.560446    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cb76c4bf-rkz24"]
Jul 06 20:50:32.560562 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.560548    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6btsj"
Jul 06 20:50:32.560650 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:50:32.560603    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token kube-api-access-tdmjl], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-cb76c4bf-rkz24" podUID="feef5889-46b1-4518-af04-2a11d9fff0c8"
Jul 06 20:50:32.602763 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.602738    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 06 20:50:32.602867 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.602783    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:50:32.602942 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.602888    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zpzbb\""
Jul 06 20:50:32.603111 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.603099    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:50:32.603164 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.603143    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 06 20:50:32.603319 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.603301    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zz997\""
Jul 06 20:50:32.603538 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.603524    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 06 20:50:32.603652 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.603562    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 06 20:50:32.603955 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.603937    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tgtzj\""
Jul 06 20:50:32.604390 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.604373    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 06 20:50:32.604514 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.604498    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-gj8vr\""
Jul 06 20:50:32.604649 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.604624    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 06 20:50:32.609391 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.609372    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-8824w"]
Jul 06 20:50:32.609857 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.609844    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 06 20:50:32.612054 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.612037    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-n7nvx"]
Jul 06 20:50:32.633758 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.633732    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/feef5889-46b1-4518-af04-2a11d9fff0c8-bound-sa-token\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.633951 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.633935    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9h8j\" (UniqueName: \"kubernetes.io/projected/af0e795f-ce9d-4463-b889-b384a89262f3-kube-api-access-r9h8j\") pod \"cluster-samples-operator-56fcf86bf-b996x\" (UID: \"af0e795f-ce9d-4463-b889-b384a89262f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b996x"
Jul 06 20:50:32.644850 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.644799    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-6btsj"]
Jul 06 20:50:32.646574 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.646540    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/61d59548-c626-4260-b1f3-4ecf218a82df-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-6btsj\" (UID: \"61d59548-c626-4260-b1f3-4ecf218a82df\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6btsj"
Jul 06 20:50:32.646834 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.646581    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf6rw\" (UniqueName: \"kubernetes.io/projected/adb77bc9-2a7a-443f-a8ca-8b119a477109-kube-api-access-hf6rw\") pod \"console-operator-68b757fb67-cxbrh\" (UID: \"adb77bc9-2a7a-443f-a8ca-8b119a477109\") " pod="openshift-console-operator/console-operator-68b757fb67-cxbrh"
Jul 06 20:50:32.646834 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.646695    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bb1a3249-22a8-4862-877b-bcaeebf12416-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-n7nvx\" (UID: \"bb1a3249-22a8-4862-877b-bcaeebf12416\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-n7nvx"
Jul 06 20:50:32.646834 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.646724    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm5cg\" (UniqueName: \"kubernetes.io/projected/ff5890ec-30ec-4004-9877-5fde2033f4d6-kube-api-access-mm5cg\") pod \"klusterlet-addon-workmgr-78d6fff86-dqqf7\" (UID: \"ff5890ec-30ec-4004-9877-5fde2033f4d6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78d6fff86-dqqf7"
Jul 06 20:50:32.646834 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.646750    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adb77bc9-2a7a-443f-a8ca-8b119a477109-trusted-ca\") pod \"console-operator-68b757fb67-cxbrh\" (UID: \"adb77bc9-2a7a-443f-a8ca-8b119a477109\") " pod="openshift-console-operator/console-operator-68b757fb67-cxbrh"
Jul 06 20:50:32.646834 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.646775    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/adb77bc9-2a7a-443f-a8ca-8b119a477109-config\") pod \"console-operator-68b757fb67-cxbrh\" (UID: \"adb77bc9-2a7a-443f-a8ca-8b119a477109\") " pod="openshift-console-operator/console-operator-68b757fb67-cxbrh"
Jul 06 20:50:32.646834 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.646792    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj9tt\" (UniqueName: \"kubernetes.io/projected/cd8dab4f-de0e-4c53-ba12-86afda4a1016-kube-api-access-kj9tt\") pod \"volume-data-source-validator-5d456fdbcd-8824w\" (UID: \"cd8dab4f-de0e-4c53-ba12-86afda4a1016\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-8824w"
Jul 06 20:50:32.646834 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.646815    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/adb77bc9-2a7a-443f-a8ca-8b119a477109-serving-cert\") pod \"console-operator-68b757fb67-cxbrh\" (UID: \"adb77bc9-2a7a-443f-a8ca-8b119a477109\") " pod="openshift-console-operator/console-operator-68b757fb67-cxbrh"
Jul 06 20:50:32.646834 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.646833    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff5890ec-30ec-4004-9877-5fde2033f4d6-tmp\") pod \"klusterlet-addon-workmgr-78d6fff86-dqqf7\" (UID: \"ff5890ec-30ec-4004-9877-5fde2033f4d6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78d6fff86-dqqf7"
Jul 06 20:50:32.647287 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.646847    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/61d59548-c626-4260-b1f3-4ecf218a82df-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-6btsj\" (UID: \"61d59548-c626-4260-b1f3-4ecf218a82df\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6btsj"
Jul 06 20:50:32.647287 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.646889    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ff5890ec-30ec-4004-9877-5fde2033f4d6-klusterlet-config\") pod \"klusterlet-addon-workmgr-78d6fff86-dqqf7\" (UID: \"ff5890ec-30ec-4004-9877-5fde2033f4d6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78d6fff86-dqqf7"
Jul 06 20:50:32.647499 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.647482    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff5890ec-30ec-4004-9877-5fde2033f4d6-tmp\") pod \"klusterlet-addon-workmgr-78d6fff86-dqqf7\" (UID: \"ff5890ec-30ec-4004-9877-5fde2033f4d6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78d6fff86-dqqf7"
Jul 06 20:50:32.649251 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.649228    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ff5890ec-30ec-4004-9877-5fde2033f4d6-klusterlet-config\") pod \"klusterlet-addon-workmgr-78d6fff86-dqqf7\" (UID: \"ff5890ec-30ec-4004-9877-5fde2033f4d6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78d6fff86-dqqf7"
Jul 06 20:50:32.651922 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.651903    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-cxbrh"]
Jul 06 20:50:32.652516 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.652490    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 06 20:50:32.653072 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.653039    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdmjl\" (UniqueName: \"kubernetes.io/projected/feef5889-46b1-4518-af04-2a11d9fff0c8-kube-api-access-tdmjl\") pod \"image-registry-cb76c4bf-rkz24\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") " pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:32.686937 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.686913    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm5cg\" (UniqueName: \"kubernetes.io/projected/ff5890ec-30ec-4004-9877-5fde2033f4d6-kube-api-access-mm5cg\") pod \"klusterlet-addon-workmgr-78d6fff86-dqqf7\" (UID: \"ff5890ec-30ec-4004-9877-5fde2033f4d6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78d6fff86-dqqf7"
Jul 06 20:50:32.699500 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.699471    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b996x"
Jul 06 20:50:32.725468 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.725443    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78d6fff86-dqqf7"
Jul 06 20:50:32.749284 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.748016    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/61d59548-c626-4260-b1f3-4ecf218a82df-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-6btsj\" (UID: \"61d59548-c626-4260-b1f3-4ecf218a82df\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6btsj"
Jul 06 20:50:32.749284 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.748111    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/61d59548-c626-4260-b1f3-4ecf218a82df-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-6btsj\" (UID: \"61d59548-c626-4260-b1f3-4ecf218a82df\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6btsj"
Jul 06 20:50:32.749284 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.748149    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf6rw\" (UniqueName: \"kubernetes.io/projected/adb77bc9-2a7a-443f-a8ca-8b119a477109-kube-api-access-hf6rw\") pod \"console-operator-68b757fb67-cxbrh\" (UID: \"adb77bc9-2a7a-443f-a8ca-8b119a477109\") " pod="openshift-console-operator/console-operator-68b757fb67-cxbrh"
Jul 06 20:50:32.749284 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.748181    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bb1a3249-22a8-4862-877b-bcaeebf12416-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-n7nvx\" (UID: \"bb1a3249-22a8-4862-877b-bcaeebf12416\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-n7nvx"
Jul 06 20:50:32.749284 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.748218    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adb77bc9-2a7a-443f-a8ca-8b119a477109-trusted-ca\") pod \"console-operator-68b757fb67-cxbrh\" (UID: \"adb77bc9-2a7a-443f-a8ca-8b119a477109\") " pod="openshift-console-operator/console-operator-68b757fb67-cxbrh"
Jul 06 20:50:32.749284 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.748254    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/adb77bc9-2a7a-443f-a8ca-8b119a477109-config\") pod \"console-operator-68b757fb67-cxbrh\" (UID: \"adb77bc9-2a7a-443f-a8ca-8b119a477109\") " pod="openshift-console-operator/console-operator-68b757fb67-cxbrh"
Jul 06 20:50:32.749284 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.748281    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj9tt\" (UniqueName: \"kubernetes.io/projected/cd8dab4f-de0e-4c53-ba12-86afda4a1016-kube-api-access-kj9tt\") pod \"volume-data-source-validator-5d456fdbcd-8824w\" (UID: \"cd8dab4f-de0e-4c53-ba12-86afda4a1016\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-8824w"
Jul 06 20:50:32.749284 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.748686    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/adb77bc9-2a7a-443f-a8ca-8b119a477109-serving-cert\") pod \"console-operator-68b757fb67-cxbrh\" (UID: \"adb77bc9-2a7a-443f-a8ca-8b119a477109\") " pod="openshift-console-operator/console-operator-68b757fb67-cxbrh"
Jul 06 20:50:32.749284 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.748763    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/61d59548-c626-4260-b1f3-4ecf218a82df-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-6btsj\" (UID: \"61d59548-c626-4260-b1f3-4ecf218a82df\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6btsj"
Jul 06 20:50:32.749764 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.749585    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adb77bc9-2a7a-443f-a8ca-8b119a477109-trusted-ca\") pod \"console-operator-68b757fb67-cxbrh\" (UID: \"adb77bc9-2a7a-443f-a8ca-8b119a477109\") " pod="openshift-console-operator/console-operator-68b757fb67-cxbrh"
Jul 06 20:50:32.750180 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.750135    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/adb77bc9-2a7a-443f-a8ca-8b119a477109-config\") pod \"console-operator-68b757fb67-cxbrh\" (UID: \"adb77bc9-2a7a-443f-a8ca-8b119a477109\") " pod="openshift-console-operator/console-operator-68b757fb67-cxbrh"
Jul 06 20:50:32.751813 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.751788    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bb1a3249-22a8-4862-877b-bcaeebf12416-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-n7nvx\" (UID: \"bb1a3249-22a8-4862-877b-bcaeebf12416\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-n7nvx"
Jul 06 20:50:32.752754 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.752731    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/61d59548-c626-4260-b1f3-4ecf218a82df-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-6btsj\" (UID: \"61d59548-c626-4260-b1f3-4ecf218a82df\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6btsj"
Jul 06 20:50:32.753184 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.753145    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/adb77bc9-2a7a-443f-a8ca-8b119a477109-serving-cert\") pod \"console-operator-68b757fb67-cxbrh\" (UID: \"adb77bc9-2a7a-443f-a8ca-8b119a477109\") " pod="openshift-console-operator/console-operator-68b757fb67-cxbrh"
Jul 06 20:50:32.770152 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.770126    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf6rw\" (UniqueName: \"kubernetes.io/projected/adb77bc9-2a7a-443f-a8ca-8b119a477109-kube-api-access-hf6rw\") pod \"console-operator-68b757fb67-cxbrh\" (UID: \"adb77bc9-2a7a-443f-a8ca-8b119a477109\") " pod="openshift-console-operator/console-operator-68b757fb67-cxbrh"
Jul 06 20:50:32.770344 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.770321    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj9tt\" (UniqueName: \"kubernetes.io/projected/cd8dab4f-de0e-4c53-ba12-86afda4a1016-kube-api-access-kj9tt\") pod \"volume-data-source-validator-5d456fdbcd-8824w\" (UID: \"cd8dab4f-de0e-4c53-ba12-86afda4a1016\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-8824w"
Jul 06 20:50:32.849331 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.849302    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b996x"]
Jul 06 20:50:32.864672 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.864647    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-78d6fff86-dqqf7"]
Jul 06 20:50:32.867115 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:50:32.867091    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5890ec_30ec_4004_9877_5fde2033f4d6.slice/crio-cd415736d99ef09f7f69e2e2298848523466823c99abf9f492d6220dc40d2dab WatchSource:0}: Error finding container cd415736d99ef09f7f69e2e2298848523466823c99abf9f492d6220dc40d2dab: Status 404 returned error can't find the container with id cd415736d99ef09f7f69e2e2298848523466823c99abf9f492d6220dc40d2dab
Jul 06 20:50:32.867968 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.867947    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-n7nvx"
Jul 06 20:50:32.873676 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.873655    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-8824w"
Jul 06 20:50:32.879415 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.879388    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-cxbrh"
Jul 06 20:50:32.884402 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.884372    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6btsj"
Jul 06 20:50:32.978377 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:32.978345    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nm7gc"
Jul 06 20:50:33.037371 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.037294    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78d6fff86-dqqf7" event={"ID":"ff5890ec-30ec-4004-9877-5fde2033f4d6","Type":"ContainerStarted","Data":"cd415736d99ef09f7f69e2e2298848523466823c99abf9f492d6220dc40d2dab"}
Jul 06 20:50:33.038711 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.038685    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:33.039242 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.038876    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b996x" event={"ID":"af0e795f-ce9d-4463-b889-b384a89262f3","Type":"ContainerStarted","Data":"3622c8bd62f67a0fc92fb51afea41e6121c1a42f9e7906652f246bb5594fb03b"}
Jul 06 20:50:33.046404 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.046339    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:33.091818 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:50:33.091792    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb1a3249_22a8_4862_877b_bcaeebf12416.slice/crio-806b58cd51c1772430d3cab2dadd8cf8f65f102a55ad075ec8ed8802a48c84e5 WatchSource:0}: Error finding container 806b58cd51c1772430d3cab2dadd8cf8f65f102a55ad075ec8ed8802a48c84e5: Status 404 returned error can't find the container with id 806b58cd51c1772430d3cab2dadd8cf8f65f102a55ad075ec8ed8802a48c84e5
Jul 06 20:50:33.092983 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:50:33.092920    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8dab4f_de0e_4c53_ba12_86afda4a1016.slice/crio-0ee6f53b4e42c0aa09d66832a85c99031e5fdfe39853555202d49e461434c51f WatchSource:0}: Error finding container 0ee6f53b4e42c0aa09d66832a85c99031e5fdfe39853555202d49e461434c51f: Status 404 returned error can't find the container with id 0ee6f53b4e42c0aa09d66832a85c99031e5fdfe39853555202d49e461434c51f
Jul 06 20:50:33.094758 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.094676    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-n7nvx"]
Jul 06 20:50:33.096701 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.096681    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-8824w"]
Jul 06 20:50:33.109757 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.109736    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-cxbrh"]
Jul 06 20:50:33.112332 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:50:33.112311    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb77bc9_2a7a_443f_a8ca_8b119a477109.slice/crio-103737e24ae6b2a1d09393af5f307f79e0d1a89138783fe65b4aaa6b1efbd60c WatchSource:0}: Error finding container 103737e24ae6b2a1d09393af5f307f79e0d1a89138783fe65b4aaa6b1efbd60c: Status 404 returned error can't find the container with id 103737e24ae6b2a1d09393af5f307f79e0d1a89138783fe65b4aaa6b1efbd60c
Jul 06 20:50:33.151641 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.151604    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/feef5889-46b1-4518-af04-2a11d9fff0c8-ca-trust-extracted\") pod \"feef5889-46b1-4518-af04-2a11d9fff0c8\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") "
Jul 06 20:50:33.151775 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.151652    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/feef5889-46b1-4518-af04-2a11d9fff0c8-bound-sa-token\") pod \"feef5889-46b1-4518-af04-2a11d9fff0c8\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") "
Jul 06 20:50:33.151775 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.151684    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/feef5889-46b1-4518-af04-2a11d9fff0c8-registry-certificates\") pod \"feef5889-46b1-4518-af04-2a11d9fff0c8\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") "
Jul 06 20:50:33.151775 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.151700    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/feef5889-46b1-4518-af04-2a11d9fff0c8-trusted-ca\") pod \"feef5889-46b1-4518-af04-2a11d9fff0c8\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") "
Jul 06 20:50:33.151775 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.151732    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/feef5889-46b1-4518-af04-2a11d9fff0c8-image-registry-private-configuration\") pod \"feef5889-46b1-4518-af04-2a11d9fff0c8\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") "
Jul 06 20:50:33.151775 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.151769    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/feef5889-46b1-4518-af04-2a11d9fff0c8-installation-pull-secrets\") pod \"feef5889-46b1-4518-af04-2a11d9fff0c8\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") "
Jul 06 20:50:33.151985 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.151804    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdmjl\" (UniqueName: \"kubernetes.io/projected/feef5889-46b1-4518-af04-2a11d9fff0c8-kube-api-access-tdmjl\") pod \"feef5889-46b1-4518-af04-2a11d9fff0c8\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") "
Jul 06 20:50:33.151985 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.151817    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feef5889-46b1-4518-af04-2a11d9fff0c8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "feef5889-46b1-4518-af04-2a11d9fff0c8" (UID: "feef5889-46b1-4518-af04-2a11d9fff0c8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:50:33.151985 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.151829    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/feef5889-46b1-4518-af04-2a11d9fff0c8-registry-tls\") pod \"feef5889-46b1-4518-af04-2a11d9fff0c8\" (UID: \"feef5889-46b1-4518-af04-2a11d9fff0c8\") "
Jul 06 20:50:33.152071 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.152054    2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/feef5889-46b1-4518-af04-2a11d9fff0c8-ca-trust-extracted\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:50:33.155055 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.155032    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feef5889-46b1-4518-af04-2a11d9fff0c8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "feef5889-46b1-4518-af04-2a11d9fff0c8" (UID: "feef5889-46b1-4518-af04-2a11d9fff0c8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:50:33.155172 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.155025    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feef5889-46b1-4518-af04-2a11d9fff0c8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "feef5889-46b1-4518-af04-2a11d9fff0c8" (UID: "feef5889-46b1-4518-af04-2a11d9fff0c8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:50:33.156819 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.156795    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feef5889-46b1-4518-af04-2a11d9fff0c8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "feef5889-46b1-4518-af04-2a11d9fff0c8" (UID: "feef5889-46b1-4518-af04-2a11d9fff0c8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:50:33.156819 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.156808    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feef5889-46b1-4518-af04-2a11d9fff0c8-kube-api-access-tdmjl" (OuterVolumeSpecName: "kube-api-access-tdmjl") pod "feef5889-46b1-4518-af04-2a11d9fff0c8" (UID: "feef5889-46b1-4518-af04-2a11d9fff0c8"). InnerVolumeSpecName "kube-api-access-tdmjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:50:33.156932 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.156858    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feef5889-46b1-4518-af04-2a11d9fff0c8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "feef5889-46b1-4518-af04-2a11d9fff0c8" (UID: "feef5889-46b1-4518-af04-2a11d9fff0c8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:50:33.157148 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.157039    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feef5889-46b1-4518-af04-2a11d9fff0c8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "feef5889-46b1-4518-af04-2a11d9fff0c8" (UID: "feef5889-46b1-4518-af04-2a11d9fff0c8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:50:33.157148 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.157060    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feef5889-46b1-4518-af04-2a11d9fff0c8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "feef5889-46b1-4518-af04-2a11d9fff0c8" (UID: "feef5889-46b1-4518-af04-2a11d9fff0c8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:50:33.252724 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.252601    2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/feef5889-46b1-4518-af04-2a11d9fff0c8-bound-sa-token\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:50:33.252724 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.252648    2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/feef5889-46b1-4518-af04-2a11d9fff0c8-registry-certificates\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:50:33.252724 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.252663    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/feef5889-46b1-4518-af04-2a11d9fff0c8-trusted-ca\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:50:33.252724 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.252676    2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/feef5889-46b1-4518-af04-2a11d9fff0c8-image-registry-private-configuration\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:50:33.252724 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.252691    2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/feef5889-46b1-4518-af04-2a11d9fff0c8-installation-pull-secrets\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:50:33.252724 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.252704    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdmjl\" (UniqueName: \"kubernetes.io/projected/feef5889-46b1-4518-af04-2a11d9fff0c8-kube-api-access-tdmjl\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:50:33.252724 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.252717    2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/feef5889-46b1-4518-af04-2a11d9fff0c8-registry-tls\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:50:33.316781 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:33.316749    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-6btsj"]
Jul 06 20:50:33.323779 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:50:33.323702    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d59548_c626_4260_b1f3_4ecf218a82df.slice/crio-9d37df9964c194095cccd859000b836d35e2c6be6b3dd585790c662fe4831501 WatchSource:0}: Error finding container 9d37df9964c194095cccd859000b836d35e2c6be6b3dd585790c662fe4831501: Status 404 returned error can't find the container with id 9d37df9964c194095cccd859000b836d35e2c6be6b3dd585790c662fe4831501
Jul 06 20:50:34.049389 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:34.049301    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-8824w" event={"ID":"cd8dab4f-de0e-4c53-ba12-86afda4a1016","Type":"ContainerStarted","Data":"0ee6f53b4e42c0aa09d66832a85c99031e5fdfe39853555202d49e461434c51f"}
Jul 06 20:50:34.052753 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:34.052667    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-cxbrh" event={"ID":"adb77bc9-2a7a-443f-a8ca-8b119a477109","Type":"ContainerStarted","Data":"103737e24ae6b2a1d09393af5f307f79e0d1a89138783fe65b4aaa6b1efbd60c"}
Jul 06 20:50:34.058419 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:34.058284    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-n7nvx" event={"ID":"bb1a3249-22a8-4862-877b-bcaeebf12416","Type":"ContainerStarted","Data":"806b58cd51c1772430d3cab2dadd8cf8f65f102a55ad075ec8ed8802a48c84e5"}
Jul 06 20:50:34.063214 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:34.062223    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb76c4bf-rkz24"
Jul 06 20:50:34.063214 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:34.062789    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6btsj" event={"ID":"61d59548-c626-4260-b1f3-4ecf218a82df","Type":"ContainerStarted","Data":"9d37df9964c194095cccd859000b836d35e2c6be6b3dd585790c662fe4831501"}
Jul 06 20:50:34.120113 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:34.120076    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cb76c4bf-rkz24"]
Jul 06 20:50:34.145244 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:34.145212    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-cb76c4bf-rkz24"]
Jul 06 20:50:35.657525 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:35.657488    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feef5889-46b1-4518-af04-2a11d9fff0c8" path="/var/lib/kubelet/pods/feef5889-46b1-4518-af04-2a11d9fff0c8/volumes"
Jul 06 20:50:38.805497 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:38.805459    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-6zzgr"]
Jul 06 20:50:38.808164 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:38.808143    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-6zzgr"
Jul 06 20:50:38.811746 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:38.811718    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 06 20:50:38.811859 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:38.811725    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 06 20:50:38.811960 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:38.811945    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-4mlmc\""
Jul 06 20:50:38.821660 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:38.821639    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-6zzgr"]
Jul 06 20:50:38.904581 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:38.904545    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lftn\" (UniqueName: \"kubernetes.io/projected/49e9374b-884a-407f-959d-47e2fcc26e12-kube-api-access-7lftn\") pod \"downloads-685bfb5f64-6zzgr\" (UID: \"49e9374b-884a-407f-959d-47e2fcc26e12\") " pod="openshift-console/downloads-685bfb5f64-6zzgr"
Jul 06 20:50:39.005382 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.005352    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lftn\" (UniqueName: \"kubernetes.io/projected/49e9374b-884a-407f-959d-47e2fcc26e12-kube-api-access-7lftn\") pod \"downloads-685bfb5f64-6zzgr\" (UID: \"49e9374b-884a-407f-959d-47e2fcc26e12\") " pod="openshift-console/downloads-685bfb5f64-6zzgr"
Jul 06 20:50:39.015883 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.015849    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lftn\" (UniqueName: \"kubernetes.io/projected/49e9374b-884a-407f-959d-47e2fcc26e12-kube-api-access-7lftn\") pod \"downloads-685bfb5f64-6zzgr\" (UID: \"49e9374b-884a-407f-959d-47e2fcc26e12\") " pod="openshift-console/downloads-685bfb5f64-6zzgr"
Jul 06 20:50:39.082293 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.082208    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-8824w" event={"ID":"cd8dab4f-de0e-4c53-ba12-86afda4a1016","Type":"ContainerStarted","Data":"a38c9cc2be251f5b9437fe1f9e64f5abc187ae01234993de5ed3ca093575b1fe"}
Jul 06 20:50:39.083914 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.083883    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b996x" event={"ID":"af0e795f-ce9d-4463-b889-b384a89262f3","Type":"ContainerStarted","Data":"4cc73f00c0e33944236905b07e304d10246e55e031ab68e5fa1f8ec1c295ec2a"}
Jul 06 20:50:39.084046 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.083932    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b996x" event={"ID":"af0e795f-ce9d-4463-b889-b384a89262f3","Type":"ContainerStarted","Data":"b325ea201bd8864af8d59a15084d41ced23e606765113e2683fe021955bdb0cf"}
Jul 06 20:50:39.085210 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.085184    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-cxbrh" event={"ID":"adb77bc9-2a7a-443f-a8ca-8b119a477109","Type":"ContainerStarted","Data":"d35e4089e4f7cd14157ee2130dba917a4e9857acd7cc757a95ef4eb414000e93"}
Jul 06 20:50:39.085358 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.085340    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-cxbrh"
Jul 06 20:50:39.086774 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.086746    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-n7nvx" event={"ID":"bb1a3249-22a8-4862-877b-bcaeebf12416","Type":"ContainerStarted","Data":"09a257f5747a5a6d7ff88906ffddd37c47cd05120459c9a5a6db5c6dfb15b3ce"}
Jul 06 20:50:39.086974 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.086915    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-n7nvx"
Jul 06 20:50:39.088348 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.088325    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78d6fff86-dqqf7" event={"ID":"ff5890ec-30ec-4004-9877-5fde2033f4d6","Type":"ContainerStarted","Data":"f2eb987798bb3bff956d92d23dbdbff887791a045d8125be6e45a6c1c462cb08"}
Jul 06 20:50:39.088500 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.088480    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78d6fff86-dqqf7"
Jul 06 20:50:39.089971 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.089950    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6btsj" event={"ID":"61d59548-c626-4260-b1f3-4ecf218a82df","Type":"ContainerStarted","Data":"208aebc423d41df4face6502df63ec2751026d8709a6a9e6064c44b440b71107"}
Jul 06 20:50:39.090070 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.090060    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78d6fff86-dqqf7"
Jul 06 20:50:39.091019 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.090998    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-cxbrh"
Jul 06 20:50:39.092017 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.092000    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-n7nvx"
Jul 06 20:50:39.104427 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.104391    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-8824w" podStartSLOduration=2.20188217 podStartE2EDuration="7.10437966s" totalImagesPullingTime="4.90249749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:39.103293208 +0000 UTC m=+87.938180573" watchObservedRunningTime="2026-07-06 20:50:39.10437966 +0000 UTC m=+87.939267024"
Jul 06 20:50:39.116429 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.116414    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-6zzgr"
Jul 06 20:50:39.120228 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.120193    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-n7nvx" podStartSLOduration=2.210448157 podStartE2EDuration="7.120182098s" totalImagesPullingTime="4.909733941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:39.119272625 +0000 UTC m=+87.954159989" watchObservedRunningTime="2026-07-06 20:50:39.120182098 +0000 UTC m=+87.955069441"
Jul 06 20:50:39.137809 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.137767    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-cxbrh" podStartSLOduration=2.241224554 podStartE2EDuration="7.137751793s" totalImagesPullingTime="4.896527239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:39.136940631 +0000 UTC m=+87.971828052" watchObservedRunningTime="2026-07-06 20:50:39.137751793 +0000 UTC m=+87.972639159"
Jul 06 20:50:39.160575 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.160522    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b996x" podStartSLOduration=2.041814494 podStartE2EDuration="7.160504907s" totalImagesPullingTime="5.118690413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:39.159710577 +0000 UTC m=+87.994597971" watchObservedRunningTime="2026-07-06 20:50:39.160504907 +0000 UTC m=+87.995392280"
Jul 06 20:50:39.180644 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.180441    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78d6fff86-dqqf7" podStartSLOduration=2.023303787 podStartE2EDuration="7.180426763s" totalImagesPullingTime="5.157122976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:39.178837807 +0000 UTC m=+88.013725198" watchObservedRunningTime="2026-07-06 20:50:39.180426763 +0000 UTC m=+88.015314131"
Jul 06 20:50:39.199435 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.198774    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6btsj" podStartSLOduration=2.515868595 podStartE2EDuration="7.198752279s" totalImagesPullingTime="4.682883684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:39.19699022 +0000 UTC m=+88.031877602" watchObservedRunningTime="2026-07-06 20:50:39.198752279 +0000 UTC m=+88.033639645"
Jul 06 20:50:39.246420 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.246398    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-6zzgr"]
Jul 06 20:50:39.248983 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:50:39.248955    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e9374b_884a_407f_959d_47e2fcc26e12.slice/crio-a5071742b833e2b76eb0c353b93ccfbbc40c0dd603ebd76876700c8213420247 WatchSource:0}: Error finding container a5071742b833e2b76eb0c353b93ccfbbc40c0dd603ebd76876700c8213420247: Status 404 returned error can't find the container with id a5071742b833e2b76eb0c353b93ccfbbc40c0dd603ebd76876700c8213420247
Jul 06 20:50:39.953061 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.953030    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-wgb8b"]
Jul 06 20:50:39.955951 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.955936    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wgb8b"
Jul 06 20:50:39.958231 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.958212    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 06 20:50:39.959049 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.959035    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-l94lx\""
Jul 06 20:50:39.959172 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.959057    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 06 20:50:39.959172 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.959095    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 06 20:50:39.968837 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:39.968818    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-wgb8b"]
Jul 06 20:50:40.013980 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:40.013940    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c05a151-03fa-4c3f-9a81-0c51e9333f54-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-wgb8b\" (UID: \"2c05a151-03fa-4c3f-9a81-0c51e9333f54\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wgb8b"
Jul 06 20:50:40.014153 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:40.013998    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c05a151-03fa-4c3f-9a81-0c51e9333f54-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-wgb8b\" (UID: \"2c05a151-03fa-4c3f-9a81-0c51e9333f54\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wgb8b"
Jul 06 20:50:40.014153 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:40.014058    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rcg9\" (UniqueName: \"kubernetes.io/projected/2c05a151-03fa-4c3f-9a81-0c51e9333f54-kube-api-access-5rcg9\") pod \"prometheus-operator-567f7b8b7c-wgb8b\" (UID: \"2c05a151-03fa-4c3f-9a81-0c51e9333f54\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wgb8b"
Jul 06 20:50:40.014153 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:40.014123    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c05a151-03fa-4c3f-9a81-0c51e9333f54-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-wgb8b\" (UID: \"2c05a151-03fa-4c3f-9a81-0c51e9333f54\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wgb8b"
Jul 06 20:50:40.096065 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:40.095505    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-6zzgr" event={"ID":"49e9374b-884a-407f-959d-47e2fcc26e12","Type":"ContainerStarted","Data":"a5071742b833e2b76eb0c353b93ccfbbc40c0dd603ebd76876700c8213420247"}
Jul 06 20:50:40.114854 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:40.114823    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c05a151-03fa-4c3f-9a81-0c51e9333f54-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-wgb8b\" (UID: \"2c05a151-03fa-4c3f-9a81-0c51e9333f54\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wgb8b"
Jul 06 20:50:40.115024 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:40.114872    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c05a151-03fa-4c3f-9a81-0c51e9333f54-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-wgb8b\" (UID: \"2c05a151-03fa-4c3f-9a81-0c51e9333f54\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wgb8b"
Jul 06 20:50:40.115024 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:40.114898    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rcg9\" (UniqueName: \"kubernetes.io/projected/2c05a151-03fa-4c3f-9a81-0c51e9333f54-kube-api-access-5rcg9\") pod \"prometheus-operator-567f7b8b7c-wgb8b\" (UID: \"2c05a151-03fa-4c3f-9a81-0c51e9333f54\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wgb8b"
Jul 06 20:50:40.115024 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:40.114928    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c05a151-03fa-4c3f-9a81-0c51e9333f54-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-wgb8b\" (UID: \"2c05a151-03fa-4c3f-9a81-0c51e9333f54\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wgb8b"
Jul 06 20:50:40.115586 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:40.115554    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c05a151-03fa-4c3f-9a81-0c51e9333f54-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-wgb8b\" (UID: \"2c05a151-03fa-4c3f-9a81-0c51e9333f54\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wgb8b"
Jul 06 20:50:40.117172 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:40.117147    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c05a151-03fa-4c3f-9a81-0c51e9333f54-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-wgb8b\" (UID: \"2c05a151-03fa-4c3f-9a81-0c51e9333f54\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wgb8b"
Jul 06 20:50:40.117258 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:40.117224    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c05a151-03fa-4c3f-9a81-0c51e9333f54-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-wgb8b\" (UID: \"2c05a151-03fa-4c3f-9a81-0c51e9333f54\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wgb8b"
Jul 06 20:50:40.124122 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:40.124102    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rcg9\" (UniqueName: \"kubernetes.io/projected/2c05a151-03fa-4c3f-9a81-0c51e9333f54-kube-api-access-5rcg9\") pod \"prometheus-operator-567f7b8b7c-wgb8b\" (UID: \"2c05a151-03fa-4c3f-9a81-0c51e9333f54\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wgb8b"
Jul 06 20:50:40.265345 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:40.265261    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wgb8b"
Jul 06 20:50:40.392527 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:40.392499    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-wgb8b"]
Jul 06 20:50:40.395550 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:50:40.395525    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c05a151_03fa_4c3f_9a81_0c51e9333f54.slice/crio-f132fa28967c1d7e35e041dac1a796a90d2cd6ee86cb703ee25919829674dfd2 WatchSource:0}: Error finding container f132fa28967c1d7e35e041dac1a796a90d2cd6ee86cb703ee25919829674dfd2: Status 404 returned error can't find the container with id f132fa28967c1d7e35e041dac1a796a90d2cd6ee86cb703ee25919829674dfd2
Jul 06 20:50:41.101120 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:41.101082    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wgb8b" event={"ID":"2c05a151-03fa-4c3f-9a81-0c51e9333f54","Type":"ContainerStarted","Data":"f132fa28967c1d7e35e041dac1a796a90d2cd6ee86cb703ee25919829674dfd2"}
Jul 06 20:50:42.108204 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:42.108162    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wgb8b" event={"ID":"2c05a151-03fa-4c3f-9a81-0c51e9333f54","Type":"ContainerStarted","Data":"aa937c75110fdfafca2fd9e4048c46da6edc87721f23f54d8ba067b6a5982228"}
Jul 06 20:50:42.108651 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:42.108229    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wgb8b" event={"ID":"2c05a151-03fa-4c3f-9a81-0c51e9333f54","Type":"ContainerStarted","Data":"c0aff3f5c202ab1123310ed7df8ce2d7020ee03f6192f6142e8d51626dbecb04"}
Jul 06 20:50:42.127770 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:42.127717    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-wgb8b" podStartSLOduration=1.990246089 podStartE2EDuration="3.127701339s" totalImagesPullingTime="1.13745525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:42.125914516 +0000 UTC m=+90.960801903" watchObservedRunningTime="2026-07-06 20:50:42.127701339 +0000 UTC m=+90.962588703"
Jul 06 20:50:44.564034 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.563999    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-jfscn"]
Jul 06 20:50:44.567754 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.567727    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-jfscn"
Jul 06 20:50:44.571129 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.571108    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 06 20:50:44.571306 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.571286    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 06 20:50:44.571983 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.571961    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5ldjt\""
Jul 06 20:50:44.589519 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.589494    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-jfscn"]
Jul 06 20:50:44.606816 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.606791    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vzht8"]
Jul 06 20:50:44.610388 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.610371    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.614833 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.614808    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4rs94\""
Jul 06 20:50:44.614933 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.614849    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 06 20:50:44.617957 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.617934    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 06 20:50:44.618267 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.617972    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 06 20:50:44.658126 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.657790    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8afe904-ec15-47fd-abf0-8b6deaa0cf4b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-jfscn\" (UID: \"d8afe904-ec15-47fd-abf0-8b6deaa0cf4b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-jfscn"
Jul 06 20:50:44.658126 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.657830    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8afe904-ec15-47fd-abf0-8b6deaa0cf4b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-jfscn\" (UID: \"d8afe904-ec15-47fd-abf0-8b6deaa0cf4b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-jfscn"
Jul 06 20:50:44.658126 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.657881    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvch8\" (UniqueName: \"kubernetes.io/projected/d8afe904-ec15-47fd-abf0-8b6deaa0cf4b-kube-api-access-nvch8\") pod \"openshift-state-metrics-54cf96dfcf-jfscn\" (UID: \"d8afe904-ec15-47fd-abf0-8b6deaa0cf4b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-jfscn"
Jul 06 20:50:44.658126 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.657983    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8afe904-ec15-47fd-abf0-8b6deaa0cf4b-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-jfscn\" (UID: \"d8afe904-ec15-47fd-abf0-8b6deaa0cf4b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-jfscn"
Jul 06 20:50:44.758481 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.758450    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-node-exporter-accelerators-collector-config\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.758481 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.758484    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtr22\" (UniqueName: \"kubernetes.io/projected/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-kube-api-access-rtr22\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.758716 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.758504    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-sys\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.758716 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.758526    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-metrics-client-ca\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.758716 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.758564    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvch8\" (UniqueName: \"kubernetes.io/projected/d8afe904-ec15-47fd-abf0-8b6deaa0cf4b-kube-api-access-nvch8\") pod \"openshift-state-metrics-54cf96dfcf-jfscn\" (UID: \"d8afe904-ec15-47fd-abf0-8b6deaa0cf4b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-jfscn"
Jul 06 20:50:44.758716 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.758586    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-node-exporter-textfile\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.758716 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.758686    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8afe904-ec15-47fd-abf0-8b6deaa0cf4b-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-jfscn\" (UID: \"d8afe904-ec15-47fd-abf0-8b6deaa0cf4b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-jfscn"
Jul 06 20:50:44.758872 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.758807    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-node-exporter-wtmp\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.758872 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.758837    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.758872 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.758868    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-root\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.758962 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.758890    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-node-exporter-tls\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.758962 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.758945    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8afe904-ec15-47fd-abf0-8b6deaa0cf4b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-jfscn\" (UID: \"d8afe904-ec15-47fd-abf0-8b6deaa0cf4b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-jfscn"
Jul 06 20:50:44.759024 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.758977    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8afe904-ec15-47fd-abf0-8b6deaa0cf4b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-jfscn\" (UID: \"d8afe904-ec15-47fd-abf0-8b6deaa0cf4b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-jfscn"
Jul 06 20:50:44.759329 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.759309    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8afe904-ec15-47fd-abf0-8b6deaa0cf4b-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-jfscn\" (UID: \"d8afe904-ec15-47fd-abf0-8b6deaa0cf4b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-jfscn"
Jul 06 20:50:44.761410 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.761385    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8afe904-ec15-47fd-abf0-8b6deaa0cf4b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-jfscn\" (UID: \"d8afe904-ec15-47fd-abf0-8b6deaa0cf4b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-jfscn"
Jul 06 20:50:44.761542 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.761525    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8afe904-ec15-47fd-abf0-8b6deaa0cf4b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-jfscn\" (UID: \"d8afe904-ec15-47fd-abf0-8b6deaa0cf4b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-jfscn"
Jul 06 20:50:44.770883 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.770861    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvch8\" (UniqueName: \"kubernetes.io/projected/d8afe904-ec15-47fd-abf0-8b6deaa0cf4b-kube-api-access-nvch8\") pod \"openshift-state-metrics-54cf96dfcf-jfscn\" (UID: \"d8afe904-ec15-47fd-abf0-8b6deaa0cf4b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-jfscn"
Jul 06 20:50:44.859869 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.859780    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-node-exporter-wtmp\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.859869 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.859819    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.859869 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.859850    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-root\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.860146 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.859873    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-node-exporter-tls\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.860146 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.859920    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-node-exporter-accelerators-collector-config\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.860146 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.859942    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtr22\" (UniqueName: \"kubernetes.io/projected/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-kube-api-access-rtr22\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.860146 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.859969    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-sys\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.860146 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.859984    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-node-exporter-wtmp\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.860146 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.859996    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-metrics-client-ca\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.860146 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.859997    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-root\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.860146 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.860053    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-sys\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.860146 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.860073    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-node-exporter-textfile\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.860795 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.860771    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-metrics-client-ca\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.860952 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.860933    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-node-exporter-textfile\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.861307 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.861223    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-node-exporter-accelerators-collector-config\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.862653 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.862582    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.862747 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.862698    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-node-exporter-tls\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.870174 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.870084    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtr22\" (UniqueName: \"kubernetes.io/projected/dc4993bc-5e06-4c35-a86a-4e74e46ac3cb-kube-api-access-rtr22\") pod \"node-exporter-vzht8\" (UID: \"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb\") " pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:44.879006 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.878968    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-jfscn"
Jul 06 20:50:44.926995 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:44.926961    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vzht8"
Jul 06 20:50:45.021713 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:45.021683    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-jfscn"]
Jul 06 20:50:45.024187 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:50:45.024161    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8afe904_ec15_47fd_abf0_8b6deaa0cf4b.slice/crio-c70e7244cb2c10ff06046c22c37c98fd7b61ddbb893e154261202360c029bff4 WatchSource:0}: Error finding container c70e7244cb2c10ff06046c22c37c98fd7b61ddbb893e154261202360c029bff4: Status 404 returned error can't find the container with id c70e7244cb2c10ff06046c22c37c98fd7b61ddbb893e154261202360c029bff4
Jul 06 20:50:45.124402 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:45.124368    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vzht8" event={"ID":"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb","Type":"ContainerStarted","Data":"feeadebad0e9edafae9e0258bc2058b41ba902646b6b004a5214170e7e24b425"}
Jul 06 20:50:45.126679 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:45.126556    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-jfscn" event={"ID":"d8afe904-ec15-47fd-abf0-8b6deaa0cf4b","Type":"ContainerStarted","Data":"d3ee3e3bc11c32100c4e1be772750543f9d9587bffdcf700d06508f521c7a175"}
Jul 06 20:50:45.126679 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:45.126640    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-jfscn" event={"ID":"d8afe904-ec15-47fd-abf0-8b6deaa0cf4b","Type":"ContainerStarted","Data":"c70e7244cb2c10ff06046c22c37c98fd7b61ddbb893e154261202360c029bff4"}
Jul 06 20:50:46.135431 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:46.135382    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-jfscn" event={"ID":"d8afe904-ec15-47fd-abf0-8b6deaa0cf4b","Type":"ContainerStarted","Data":"adb4cdad38d6aed617e1ae7e216396f7880c252c01148c941099c776a974ca2f"}
Jul 06 20:50:47.142663 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.142495    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-jfscn" event={"ID":"d8afe904-ec15-47fd-abf0-8b6deaa0cf4b","Type":"ContainerStarted","Data":"d452b32dfe61352b3b004f9e1de1b8885befb9f852cbe87712c9a3c5b680a1bf"}
Jul 06 20:50:47.144378 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.144354    2559 generic.go:356] "Generic (PLEG): container finished" podID="dc4993bc-5e06-4c35-a86a-4e74e46ac3cb" containerID="b689840934d64b82feb2a855311e903670a6c8d1ad56541a143d5a85121996a9" exitCode=0
Jul 06 20:50:47.144486 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.144417    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vzht8" event={"ID":"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb","Type":"ContainerDied","Data":"b689840934d64b82feb2a855311e903670a6c8d1ad56541a143d5a85121996a9"}
Jul 06 20:50:47.167798 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.167744    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-jfscn" podStartSLOduration=2.223916481 podStartE2EDuration="3.167723157s" totalImagesPullingTime="943.806676ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:47.166204095 +0000 UTC m=+96.001091479" watchObservedRunningTime="2026-07-06 20:50:47.167723157 +0000 UTC m=+96.002610523"
Jul 06 20:50:47.585817 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.585783    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-747bf868c5-jpjjz"]
Jul 06 20:50:47.593796 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.593767    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.596270 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.596240    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 06 20:50:47.596270 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.596263    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 06 20:50:47.596668 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.596649    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-dnfhn\""
Jul 06 20:50:47.596862 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.596840    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-73fan7ec4iffq\""
Jul 06 20:50:47.596974 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.596955    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 06 20:50:47.598378 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.598355    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 06 20:50:47.598889 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.598650    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 06 20:50:47.606799 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.606779    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-747bf868c5-jpjjz"]
Jul 06 20:50:47.686890 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.686857    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbbn5\" (UniqueName: \"kubernetes.io/projected/2e837295-2af1-4f36-a83d-eba9074cc1a4-kube-api-access-bbbn5\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.687075 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.686933    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e837295-2af1-4f36-a83d-eba9074cc1a4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.687075 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.686965    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2e837295-2af1-4f36-a83d-eba9074cc1a4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.687075 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.687008    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e837295-2af1-4f36-a83d-eba9074cc1a4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.687243 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.687079    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2e837295-2af1-4f36-a83d-eba9074cc1a4-secret-grpc-tls\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.687243 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.687104    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2e837295-2af1-4f36-a83d-eba9074cc1a4-secret-thanos-querier-tls\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.687243 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.687137    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2e837295-2af1-4f36-a83d-eba9074cc1a4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.687243 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.687167    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e837295-2af1-4f36-a83d-eba9074cc1a4-metrics-client-ca\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.788246 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.788212    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e837295-2af1-4f36-a83d-eba9074cc1a4-metrics-client-ca\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.788246 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.788251    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbbn5\" (UniqueName: \"kubernetes.io/projected/2e837295-2af1-4f36-a83d-eba9074cc1a4-kube-api-access-bbbn5\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.788445 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.788310    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e837295-2af1-4f36-a83d-eba9074cc1a4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.788445 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.788336    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2e837295-2af1-4f36-a83d-eba9074cc1a4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.788445 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.788383    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e837295-2af1-4f36-a83d-eba9074cc1a4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.788445 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.788421    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2e837295-2af1-4f36-a83d-eba9074cc1a4-secret-grpc-tls\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.788663 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.788449    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2e837295-2af1-4f36-a83d-eba9074cc1a4-secret-thanos-querier-tls\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.788663 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.788481    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2e837295-2af1-4f36-a83d-eba9074cc1a4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.789126 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.789099    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e837295-2af1-4f36-a83d-eba9074cc1a4-metrics-client-ca\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.791322 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.791293    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2e837295-2af1-4f36-a83d-eba9074cc1a4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.791440 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.791421    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2e837295-2af1-4f36-a83d-eba9074cc1a4-secret-grpc-tls\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.791539 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.791516    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2e837295-2af1-4f36-a83d-eba9074cc1a4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.791578 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.791546    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e837295-2af1-4f36-a83d-eba9074cc1a4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.791747 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.791730    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2e837295-2af1-4f36-a83d-eba9074cc1a4-secret-thanos-querier-tls\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.791795 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.791729    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e837295-2af1-4f36-a83d-eba9074cc1a4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.805637 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.805598    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbbn5\" (UniqueName: \"kubernetes.io/projected/2e837295-2af1-4f36-a83d-eba9074cc1a4-kube-api-access-bbbn5\") pod \"thanos-querier-747bf868c5-jpjjz\" (UID: \"2e837295-2af1-4f36-a83d-eba9074cc1a4\") " pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:47.904861 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:47.904777    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:50:48.045950 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:48.045924    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-747bf868c5-jpjjz"]
Jul 06 20:50:48.048100 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:50:48.048057    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e837295_2af1_4f36_a83d_eba9074cc1a4.slice/crio-2c33abeaca8d7928498eb249310ba5b03c7f3dedae8fd1e0ffc3bea37f47245b WatchSource:0}: Error finding container 2c33abeaca8d7928498eb249310ba5b03c7f3dedae8fd1e0ffc3bea37f47245b: Status 404 returned error can't find the container with id 2c33abeaca8d7928498eb249310ba5b03c7f3dedae8fd1e0ffc3bea37f47245b
Jul 06 20:50:48.149758 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:48.149723    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz" event={"ID":"2e837295-2af1-4f36-a83d-eba9074cc1a4","Type":"ContainerStarted","Data":"2c33abeaca8d7928498eb249310ba5b03c7f3dedae8fd1e0ffc3bea37f47245b"}
Jul 06 20:50:48.152272 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:48.152243    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vzht8" event={"ID":"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb","Type":"ContainerStarted","Data":"81379d1ccdd230dcbb369999f1c4efbf284d05dc31cd5b61562a3869ebd887ff"}
Jul 06 20:50:48.152386 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:48.152300    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vzht8" event={"ID":"dc4993bc-5e06-4c35-a86a-4e74e46ac3cb","Type":"ContainerStarted","Data":"672b3941b73604469c1681b3bba75e86f4358904d580ffb85cb36b819028be42"}
Jul 06 20:50:48.175813 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:48.175719    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vzht8" podStartSLOduration=2.945466283 podStartE2EDuration="4.17569868s" totalImagesPullingTime="1.156179469s" totalInitContainerRuntime="74.052928ms" isStatefulPod=true podCreationTimestamp="2026-07-06 20:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:48.172786161 +0000 UTC m=+97.007673554" watchObservedRunningTime="2026-07-06 20:50:48.17569868 +0000 UTC m=+97.010586048"
Jul 06 20:50:49.045351 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.045291    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"]
Jul 06 20:50:49.049001 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.048977    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.052677 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.052534    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 06 20:50:49.052677 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.052672    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 06 20:50:49.052677 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.052699    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 06 20:50:49.053031 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.052744    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 06 20:50:49.053031 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.052843    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2t9q4\""
Jul 06 20:50:49.053031 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.053002    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4idvcnlk7313s\""
Jul 06 20:50:49.068509 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.065810    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"]
Jul 06 20:50:49.202062 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.202013    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/41b6c629-1831-4ee6-a0c6-5fe497398a94-client-ca-bundle\") pod \"metrics-server-7cc4d4799f-rqsdn\" (UID: \"41b6c629-1831-4ee6-a0c6-5fe497398a94\") " pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.202670 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.202068    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/41b6c629-1831-4ee6-a0c6-5fe497398a94-secret-metrics-server-tls\") pod \"metrics-server-7cc4d4799f-rqsdn\" (UID: \"41b6c629-1831-4ee6-a0c6-5fe497398a94\") " pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.202670 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.202105    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41b6c629-1831-4ee6-a0c6-5fe497398a94-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cc4d4799f-rqsdn\" (UID: \"41b6c629-1831-4ee6-a0c6-5fe497398a94\") " pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.202670 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.202141    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/41b6c629-1831-4ee6-a0c6-5fe497398a94-audit-log\") pod \"metrics-server-7cc4d4799f-rqsdn\" (UID: \"41b6c629-1831-4ee6-a0c6-5fe497398a94\") " pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.202670 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.202278    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/41b6c629-1831-4ee6-a0c6-5fe497398a94-metrics-server-audit-profiles\") pod \"metrics-server-7cc4d4799f-rqsdn\" (UID: \"41b6c629-1831-4ee6-a0c6-5fe497398a94\") " pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.202670 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.202332    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/41b6c629-1831-4ee6-a0c6-5fe497398a94-secret-metrics-server-client-certs\") pod \"metrics-server-7cc4d4799f-rqsdn\" (UID: \"41b6c629-1831-4ee6-a0c6-5fe497398a94\") " pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.202670 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.202416    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjkfk\" (UniqueName: \"kubernetes.io/projected/41b6c629-1831-4ee6-a0c6-5fe497398a94-kube-api-access-wjkfk\") pod \"metrics-server-7cc4d4799f-rqsdn\" (UID: \"41b6c629-1831-4ee6-a0c6-5fe497398a94\") " pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.303645 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.303530    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/41b6c629-1831-4ee6-a0c6-5fe497398a94-metrics-server-audit-profiles\") pod \"metrics-server-7cc4d4799f-rqsdn\" (UID: \"41b6c629-1831-4ee6-a0c6-5fe497398a94\") " pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.303809 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.303676    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/41b6c629-1831-4ee6-a0c6-5fe497398a94-secret-metrics-server-client-certs\") pod \"metrics-server-7cc4d4799f-rqsdn\" (UID: \"41b6c629-1831-4ee6-a0c6-5fe497398a94\") " pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.303809 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.303753    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjkfk\" (UniqueName: \"kubernetes.io/projected/41b6c629-1831-4ee6-a0c6-5fe497398a94-kube-api-access-wjkfk\") pod \"metrics-server-7cc4d4799f-rqsdn\" (UID: \"41b6c629-1831-4ee6-a0c6-5fe497398a94\") " pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.303929 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.303811    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/41b6c629-1831-4ee6-a0c6-5fe497398a94-client-ca-bundle\") pod \"metrics-server-7cc4d4799f-rqsdn\" (UID: \"41b6c629-1831-4ee6-a0c6-5fe497398a94\") " pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.303929 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.303847    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/41b6c629-1831-4ee6-a0c6-5fe497398a94-secret-metrics-server-tls\") pod \"metrics-server-7cc4d4799f-rqsdn\" (UID: \"41b6c629-1831-4ee6-a0c6-5fe497398a94\") " pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.303929 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.303876    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41b6c629-1831-4ee6-a0c6-5fe497398a94-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cc4d4799f-rqsdn\" (UID: \"41b6c629-1831-4ee6-a0c6-5fe497398a94\") " pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.303929 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.303901    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/41b6c629-1831-4ee6-a0c6-5fe497398a94-audit-log\") pod \"metrics-server-7cc4d4799f-rqsdn\" (UID: \"41b6c629-1831-4ee6-a0c6-5fe497398a94\") " pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.305646 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.304655    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/41b6c629-1831-4ee6-a0c6-5fe497398a94-metrics-server-audit-profiles\") pod \"metrics-server-7cc4d4799f-rqsdn\" (UID: \"41b6c629-1831-4ee6-a0c6-5fe497398a94\") " pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.305646 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.304865    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41b6c629-1831-4ee6-a0c6-5fe497398a94-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cc4d4799f-rqsdn\" (UID: \"41b6c629-1831-4ee6-a0c6-5fe497398a94\") " pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.305646 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.305153    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/41b6c629-1831-4ee6-a0c6-5fe497398a94-audit-log\") pod \"metrics-server-7cc4d4799f-rqsdn\" (UID: \"41b6c629-1831-4ee6-a0c6-5fe497398a94\") " pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.308334 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.308173    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/41b6c629-1831-4ee6-a0c6-5fe497398a94-secret-metrics-server-tls\") pod \"metrics-server-7cc4d4799f-rqsdn\" (UID: \"41b6c629-1831-4ee6-a0c6-5fe497398a94\") " pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.308461 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.308357    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/41b6c629-1831-4ee6-a0c6-5fe497398a94-secret-metrics-server-client-certs\") pod \"metrics-server-7cc4d4799f-rqsdn\" (UID: \"41b6c629-1831-4ee6-a0c6-5fe497398a94\") " pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.308577 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.308270    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/41b6c629-1831-4ee6-a0c6-5fe497398a94-client-ca-bundle\") pod \"metrics-server-7cc4d4799f-rqsdn\" (UID: \"41b6c629-1831-4ee6-a0c6-5fe497398a94\") " pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.313602 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.313566    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjkfk\" (UniqueName: \"kubernetes.io/projected/41b6c629-1831-4ee6-a0c6-5fe497398a94-kube-api-access-wjkfk\") pod \"metrics-server-7cc4d4799f-rqsdn\" (UID: \"41b6c629-1831-4ee6-a0c6-5fe497398a94\") " pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.361930 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.361705    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:50:49.978526 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:49.978498    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"]
Jul 06 20:50:49.988775 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:50:49.988736    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b6c629_1831_4ee6_a0c6_5fe497398a94.slice/crio-8286e66c49a99088eb7a86d5841682440fa22e750e65ffcbe9baaa9e19eac7cc WatchSource:0}: Error finding container 8286e66c49a99088eb7a86d5841682440fa22e750e65ffcbe9baaa9e19eac7cc: Status 404 returned error can't find the container with id 8286e66c49a99088eb7a86d5841682440fa22e750e65ffcbe9baaa9e19eac7cc
Jul 06 20:50:50.164998 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.164943    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn" event={"ID":"41b6c629-1831-4ee6-a0c6-5fe497398a94","Type":"ContainerStarted","Data":"8286e66c49a99088eb7a86d5841682440fa22e750e65ffcbe9baaa9e19eac7cc"}
Jul 06 20:50:50.167453 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.167414    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz" event={"ID":"2e837295-2af1-4f36-a83d-eba9074cc1a4","Type":"ContainerStarted","Data":"dfaee8f9c293ed5bb4707d436825ba5406a1cbb60feb34f5374cf9242dce619e"}
Jul 06 20:50:50.167587 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.167489    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz" event={"ID":"2e837295-2af1-4f36-a83d-eba9074cc1a4","Type":"ContainerStarted","Data":"806c53980bad2d234c138c5a1905190cf56bd2fc4b7b6e7773a638f3702aa3e9"}
Jul 06 20:50:50.167587 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.167531    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz" event={"ID":"2e837295-2af1-4f36-a83d-eba9074cc1a4","Type":"ContainerStarted","Data":"7e12ba4fcaa20745d3b9daab942f66ee8ba19b5cb81711b05bc0c2d1913440c2"}
Jul 06 20:50:50.927099 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.927061    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:50:50.933710 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.933684    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:50.939457 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.937133    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 06 20:50:50.939457 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.937372    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 06 20:50:50.939457 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.937551    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 06 20:50:50.939457 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.937870    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 06 20:50:50.939457 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.938057    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 06 20:50:50.939457 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.938213    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 06 20:50:50.939457 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.938381    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bj4p8s0b8opik\""
Jul 06 20:50:50.939457 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.938590    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 06 20:50:50.939457 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.938814    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x6w6j\""
Jul 06 20:50:50.939457 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.938970    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 06 20:50:50.956362 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.956337    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 06 20:50:50.956984 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.956958    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 06 20:50:50.959510 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.958397    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 06 20:50:50.959510 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.958584    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 06 20:50:50.978482 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.978391    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:50:50.980327 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.980114    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 06 20:50:50.982680 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:50.982548    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 06 20:50:51.023584 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.022986    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.023584 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.023058    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.023584 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.023085    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.023584 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.023115    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.023584 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.023144    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/47bd3e07-5f95-4203-a46f-23e5819cc710-config-out\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.023584 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.023170    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.023584 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.023195    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.023584 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.023229    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-config\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.023584 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.023255    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.023584 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.023277    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rz9s\" (UniqueName: \"kubernetes.io/projected/47bd3e07-5f95-4203-a46f-23e5819cc710-kube-api-access-8rz9s\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.023584 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.023305    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.023584 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.023327    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.023584 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.023375    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-web-config\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.023584 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.023401    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/47bd3e07-5f95-4203-a46f-23e5819cc710-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.023584 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.023420    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.023584 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.023443    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.024373 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.023472    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.024373 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.023712    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.024373 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.023752    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.024373 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.023798    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.125222 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.125191    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.126146 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.126078    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.126782 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.126756    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-web-config\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.126891 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.126823    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/47bd3e07-5f95-4203-a46f-23e5819cc710-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.126891 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.126849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.126996 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.126901    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.126996 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.126932    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.127089 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.127003    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.127089 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.127056    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.127188 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.127089    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.127188 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.127155    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.127320 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.127303    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.127372 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.127357    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.127477 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.127392    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.127477 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.127434    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/47bd3e07-5f95-4203-a46f-23e5819cc710-config-out\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.127576 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.127565    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.127651 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.127597    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.127704 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.127669    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-config\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.129104 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.129030    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.130149 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.129084    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rz9s\" (UniqueName: \"kubernetes.io/projected/47bd3e07-5f95-4203-a46f-23e5819cc710-kube-api-access-8rz9s\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.130149 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.129385    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.130149 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.129454    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.130149 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.129695    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.130149 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.129740    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.134773 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.134399    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.136912 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.135296    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.136912 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.135806    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.136912 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.136240    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.136912 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.136296    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-web-config\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.136912 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.136725    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.136912 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.136837    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.143397 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.137660    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.143397 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.139256    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/47bd3e07-5f95-4203-a46f-23e5819cc710-config-out\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.143547 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.143407    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.145395 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.144001    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.145395 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.145332    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-config\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.146828 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.146568    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.147354 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.147333    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.147427 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.147382    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/47bd3e07-5f95-4203-a46f-23e5819cc710-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.176154 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.176077    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rz9s\" (UniqueName: \"kubernetes.io/projected/47bd3e07-5f95-4203-a46f-23e5819cc710-kube-api-access-8rz9s\") pod \"prometheus-k8s-0\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:51.182083 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.182030    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz" event={"ID":"2e837295-2af1-4f36-a83d-eba9074cc1a4","Type":"ContainerStarted","Data":"761b71e6a5e3507f1827364333027a5d8b91b542f258d291726ff795880056c3"}
Jul 06 20:50:51.182217 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.182141    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz" event={"ID":"2e837295-2af1-4f36-a83d-eba9074cc1a4","Type":"ContainerStarted","Data":"ea2039dfbe5d48ee3ef36c2aa69ca6530a183adf40a84c0b552f20879d067920"}
Jul 06 20:50:51.248790 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:51.248697    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:50:53.978980 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:53.978944    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-p9bcs"
Jul 06 20:50:59.449538 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:50:59.449478    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:50:59.457581 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:50:59.457554    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47bd3e07_5f95_4203_a46f_23e5819cc710.slice/crio-01883a606c73b07d1a8c50a28ea8c82ad5f1573f3a4a954343bd1c556a87754f WatchSource:0}: Error finding container 01883a606c73b07d1a8c50a28ea8c82ad5f1573f3a4a954343bd1c556a87754f: Status 404 returned error can't find the container with id 01883a606c73b07d1a8c50a28ea8c82ad5f1573f3a4a954343bd1c556a87754f
Jul 06 20:51:00.227446 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:00.227406    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz" event={"ID":"2e837295-2af1-4f36-a83d-eba9074cc1a4","Type":"ContainerStarted","Data":"fc46dfef747f2f470f52f76b0482f63b5f3ea577d4b71f12c949986adbed7637"}
Jul 06 20:51:00.227806 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:00.227786    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:51:00.236702 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:00.236354    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn" event={"ID":"41b6c629-1831-4ee6-a0c6-5fe497398a94","Type":"ContainerStarted","Data":"17256ba7cd7b89cc6c42b6af4b30db1e61f8fcbbf18ec2426780ebfa197d5caa"}
Jul 06 20:51:00.237732 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:00.237661    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz"
Jul 06 20:51:00.241342 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:00.239274    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-6zzgr" event={"ID":"49e9374b-884a-407f-959d-47e2fcc26e12","Type":"ContainerStarted","Data":"1f7b7ceda4d1a41da805322cc08b325229e0bb09286c98971817627311aa6ec1"}
Jul 06 20:51:00.241342 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:00.240075    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-6zzgr"
Jul 06 20:51:00.241905 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:00.241882    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47bd3e07-5f95-4203-a46f-23e5819cc710","Type":"ContainerStarted","Data":"01883a606c73b07d1a8c50a28ea8c82ad5f1573f3a4a954343bd1c556a87754f"}
Jul 06 20:51:00.255065 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:00.254837    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-6zzgr"
Jul 06 20:51:00.300033 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:00.299973    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn" podStartSLOduration=1.987862549 podStartE2EDuration="11.299949474s" totalImagesPullingTime="9.312086925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:51:00.298439252 +0000 UTC m=+109.133326636" watchObservedRunningTime="2026-07-06 20:51:00.299949474 +0000 UTC m=+109.134836840"
Jul 06 20:51:00.300362 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:00.300321    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-747bf868c5-jpjjz" podStartSLOduration=10.618971872 podStartE2EDuration="13.300311878s" totalImagesPullingTime="2.681340006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:51:00.26514866 +0000 UTC m=+109.100036046" watchObservedRunningTime="2026-07-06 20:51:00.300311878 +0000 UTC m=+109.135199243"
Jul 06 20:51:00.320923 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:00.320874    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-6zzgr" podStartSLOduration=2.203681828 podStartE2EDuration="22.320857822s" totalImagesPullingTime="20.117175994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:51:00.320294088 +0000 UTC m=+109.155181454" watchObservedRunningTime="2026-07-06 20:51:00.320857822 +0000 UTC m=+109.155745188"
Jul 06 20:51:02.253200 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:02.253162    2559 generic.go:356] "Generic (PLEG): container finished" podID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerID="d85896a766d676947ad3e1512fa22c5ac9431aec743d28ec0d6b28885ed1be23" exitCode=0
Jul 06 20:51:02.253684 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:02.253256    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47bd3e07-5f95-4203-a46f-23e5819cc710","Type":"ContainerDied","Data":"d85896a766d676947ad3e1512fa22c5ac9431aec743d28ec0d6b28885ed1be23"}
Jul 06 20:51:06.276456 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:06.276422    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47bd3e07-5f95-4203-a46f-23e5819cc710","Type":"ContainerStarted","Data":"f239b9f6ca4ac9b6c581b2af849b24008480b8f664ef996f4801a994979bd7cb"}
Jul 06 20:51:07.285061 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:07.285021    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47bd3e07-5f95-4203-a46f-23e5819cc710","Type":"ContainerStarted","Data":"75dbf22d340e6be6220b2292149c7937d4579197149c3ac2cdabcbdf81ddb4c6"}
Jul 06 20:51:07.285547 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:07.285130    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47bd3e07-5f95-4203-a46f-23e5819cc710","Type":"ContainerStarted","Data":"838593a0a3cdc964f8d184b2deac8aa20a14d23873be3ee0b9cfb72e003cb781"}
Jul 06 20:51:07.285547 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:07.285176    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47bd3e07-5f95-4203-a46f-23e5819cc710","Type":"ContainerStarted","Data":"38b78d7a3456af84a585e6aae104893d4590ac75284cd8e71d45ec903d70b382"}
Jul 06 20:51:07.285547 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:07.285214    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47bd3e07-5f95-4203-a46f-23e5819cc710","Type":"ContainerStarted","Data":"8793af8b711b9956e838fd8694c991cc7a056b44ae9e4bb1dc19651f6df9a09e"}
Jul 06 20:51:07.285547 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:07.285239    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47bd3e07-5f95-4203-a46f-23e5819cc710","Type":"ContainerStarted","Data":"7078160252b47bfcdb2b059b86f164cd624a4e3534ddb197b54d6a9faf055664"}
Jul 06 20:51:07.324883 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:07.324824    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=11.680260415 podStartE2EDuration="17.324803964s" totalImagesPullingTime="5.62793866s" totalInitContainerRuntime="16.604889ms" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:51:07.322193495 +0000 UTC m=+116.157080861" watchObservedRunningTime="2026-07-06 20:51:07.324803964 +0000 UTC m=+116.159691329"
Jul 06 20:51:09.362560 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:09.362526    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:51:09.363066 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:09.362604    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:51:11.249658 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:11.249602    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:51:18.326858 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:18.326828    2559 generic.go:356] "Generic (PLEG): container finished" podID="d43e04c1-782e-409e-a666-d1e2994fb8a5" containerID="af37d7ffee4c3025d079e7b8ec68127822e1faa3b130b0cddaf823dc02ac940c" exitCode=0
Jul 06 20:51:18.327278 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:18.326888    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-h89pn" event={"ID":"d43e04c1-782e-409e-a666-d1e2994fb8a5","Type":"ContainerDied","Data":"af37d7ffee4c3025d079e7b8ec68127822e1faa3b130b0cddaf823dc02ac940c"}
Jul 06 20:51:18.327278 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:18.327176    2559 scope.go:117] "RemoveContainer" containerID="af37d7ffee4c3025d079e7b8ec68127822e1faa3b130b0cddaf823dc02ac940c"
Jul 06 20:51:19.332135 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:19.332101    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-h89pn" event={"ID":"d43e04c1-782e-409e-a666-d1e2994fb8a5","Type":"ContainerStarted","Data":"e43c69cd5f6cb11a05845a330e4e0069e46c4356c1fe4f879ccab84d4f451893"}
Jul 06 20:51:21.417181 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:21.417148    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs\") pod \"network-metrics-daemon-ckj9h\" (UID: \"ad24b800-9783-47dc-aaa1-4d2bbc01afb8\") " pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:51:21.419336 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:21.419319    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad24b800-9783-47dc-aaa1-4d2bbc01afb8-metrics-certs\") pod \"network-metrics-daemon-ckj9h\" (UID: \"ad24b800-9783-47dc-aaa1-4d2bbc01afb8\") " pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:51:21.671532 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:21.671461    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mf9wp\""
Jul 06 20:51:21.679885 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:21.679867    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckj9h"
Jul 06 20:51:21.828722 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:21.828702    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ckj9h"]
Jul 06 20:51:21.830462 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:51:21.830428    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24b800_9783_47dc_aaa1_4d2bbc01afb8.slice/crio-b631c4730cb2e344c0bfafbd524126895f0bde8f9f5639c854d534490b6651a5 WatchSource:0}: Error finding container b631c4730cb2e344c0bfafbd524126895f0bde8f9f5639c854d534490b6651a5: Status 404 returned error can't find the container with id b631c4730cb2e344c0bfafbd524126895f0bde8f9f5639c854d534490b6651a5
Jul 06 20:51:22.346838 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:22.346796    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ckj9h" event={"ID":"ad24b800-9783-47dc-aaa1-4d2bbc01afb8","Type":"ContainerStarted","Data":"b631c4730cb2e344c0bfafbd524126895f0bde8f9f5639c854d534490b6651a5"}
Jul 06 20:51:23.352535 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:23.352511    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ckj9h" event={"ID":"ad24b800-9783-47dc-aaa1-4d2bbc01afb8","Type":"ContainerStarted","Data":"66e6bc9ca345e1670aa07dae66a49ba84d85a2e56596d6ede1a3e50c4cd55f8c"}
Jul 06 20:51:24.360222 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:24.359519    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ckj9h" event={"ID":"ad24b800-9783-47dc-aaa1-4d2bbc01afb8","Type":"ContainerStarted","Data":"840027e2c49608dce8248b1942737af4f33559bc002ff214a02a21376967ef12"}
Jul 06 20:51:24.381708 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:24.381665    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ckj9h" podStartSLOduration=131.031731703 podStartE2EDuration="2m12.381649388s" totalImagesPullingTime="1.349917685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:51:24.379114914 +0000 UTC m=+133.214002289" watchObservedRunningTime="2026-07-06 20:51:24.381649388 +0000 UTC m=+133.216536753"
Jul 06 20:51:29.368150 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:29.368121    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:51:29.372599 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:29.372577    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7cc4d4799f-rqsdn"
Jul 06 20:51:51.249666 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:51.249556    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:51:51.312006 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:51.311978    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:51:51.484993 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:51:51.484968    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:09.321683 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.321591    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:52:09.322597 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.322559    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="prometheus" containerID="cri-o://f239b9f6ca4ac9b6c581b2af849b24008480b8f664ef996f4801a994979bd7cb" gracePeriod=600
Jul 06 20:52:09.322750 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.322598    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="kube-rbac-proxy-thanos" containerID="cri-o://75dbf22d340e6be6220b2292149c7937d4579197149c3ac2cdabcbdf81ddb4c6" gracePeriod=600
Jul 06 20:52:09.322750 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.322641    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="kube-rbac-proxy-web" containerID="cri-o://38b78d7a3456af84a585e6aae104893d4590ac75284cd8e71d45ec903d70b382" gracePeriod=600
Jul 06 20:52:09.322750 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.322674    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="thanos-sidecar" containerID="cri-o://8793af8b711b9956e838fd8694c991cc7a056b44ae9e4bb1dc19651f6df9a09e" gracePeriod=600
Jul 06 20:52:09.322750 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.322729    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="kube-rbac-proxy" containerID="cri-o://838593a0a3cdc964f8d184b2deac8aa20a14d23873be3ee0b9cfb72e003cb781" gracePeriod=600
Jul 06 20:52:09.322980 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.322717    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="config-reloader" containerID="cri-o://7078160252b47bfcdb2b059b86f164cd624a4e3534ddb197b54d6a9faf055664" gracePeriod=600
Jul 06 20:52:09.551484 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.551436    2559 generic.go:356] "Generic (PLEG): container finished" podID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerID="75dbf22d340e6be6220b2292149c7937d4579197149c3ac2cdabcbdf81ddb4c6" exitCode=0
Jul 06 20:52:09.551484 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.551475    2559 generic.go:356] "Generic (PLEG): container finished" podID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerID="838593a0a3cdc964f8d184b2deac8aa20a14d23873be3ee0b9cfb72e003cb781" exitCode=0
Jul 06 20:52:09.551484 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.551482    2559 generic.go:356] "Generic (PLEG): container finished" podID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerID="38b78d7a3456af84a585e6aae104893d4590ac75284cd8e71d45ec903d70b382" exitCode=0
Jul 06 20:52:09.551484 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.551487    2559 generic.go:356] "Generic (PLEG): container finished" podID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerID="8793af8b711b9956e838fd8694c991cc7a056b44ae9e4bb1dc19651f6df9a09e" exitCode=0
Jul 06 20:52:09.551484 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.551492    2559 generic.go:356] "Generic (PLEG): container finished" podID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerID="7078160252b47bfcdb2b059b86f164cd624a4e3534ddb197b54d6a9faf055664" exitCode=0
Jul 06 20:52:09.551812 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.551497    2559 generic.go:356] "Generic (PLEG): container finished" podID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerID="f239b9f6ca4ac9b6c581b2af849b24008480b8f664ef996f4801a994979bd7cb" exitCode=0
Jul 06 20:52:09.551812 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.551493    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47bd3e07-5f95-4203-a46f-23e5819cc710","Type":"ContainerDied","Data":"75dbf22d340e6be6220b2292149c7937d4579197149c3ac2cdabcbdf81ddb4c6"}
Jul 06 20:52:09.551812 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.551578    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47bd3e07-5f95-4203-a46f-23e5819cc710","Type":"ContainerDied","Data":"838593a0a3cdc964f8d184b2deac8aa20a14d23873be3ee0b9cfb72e003cb781"}
Jul 06 20:52:09.551812 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.551641    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47bd3e07-5f95-4203-a46f-23e5819cc710","Type":"ContainerDied","Data":"38b78d7a3456af84a585e6aae104893d4590ac75284cd8e71d45ec903d70b382"}
Jul 06 20:52:09.551812 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.551683    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47bd3e07-5f95-4203-a46f-23e5819cc710","Type":"ContainerDied","Data":"8793af8b711b9956e838fd8694c991cc7a056b44ae9e4bb1dc19651f6df9a09e"}
Jul 06 20:52:09.551812 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.551732    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47bd3e07-5f95-4203-a46f-23e5819cc710","Type":"ContainerDied","Data":"7078160252b47bfcdb2b059b86f164cd624a4e3534ddb197b54d6a9faf055664"}
Jul 06 20:52:09.551812 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.551770    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47bd3e07-5f95-4203-a46f-23e5819cc710","Type":"ContainerDied","Data":"f239b9f6ca4ac9b6c581b2af849b24008480b8f664ef996f4801a994979bd7cb"}
Jul 06 20:52:09.567957 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.566670    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:09.632949 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.632864    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-thanos-prometheus-http-client-file\") pod \"47bd3e07-5f95-4203-a46f-23e5819cc710\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") "
Jul 06 20:52:09.632949 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.632912    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rz9s\" (UniqueName: \"kubernetes.io/projected/47bd3e07-5f95-4203-a46f-23e5819cc710-kube-api-access-8rz9s\") pod \"47bd3e07-5f95-4203-a46f-23e5819cc710\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") "
Jul 06 20:52:09.633153 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.632954    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"47bd3e07-5f95-4203-a46f-23e5819cc710\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") "
Jul 06 20:52:09.633153 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.632996    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-kube-rbac-proxy\") pod \"47bd3e07-5f95-4203-a46f-23e5819cc710\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") "
Jul 06 20:52:09.633153 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.633021    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-rulefiles-0\") pod \"47bd3e07-5f95-4203-a46f-23e5819cc710\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") "
Jul 06 20:52:09.633475 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.633452    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-configmap-serving-certs-ca-bundle\") pod \"47bd3e07-5f95-4203-a46f-23e5819cc710\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") "
Jul 06 20:52:09.633568 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.633500    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-metrics-client-certs\") pod \"47bd3e07-5f95-4203-a46f-23e5819cc710\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") "
Jul 06 20:52:09.633568 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.633542    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-configmap-kubelet-serving-ca-bundle\") pod \"47bd3e07-5f95-4203-a46f-23e5819cc710\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") "
Jul 06 20:52:09.633695 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.633574    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/47bd3e07-5f95-4203-a46f-23e5819cc710-config-out\") pod \"47bd3e07-5f95-4203-a46f-23e5819cc710\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") "
Jul 06 20:52:09.633695 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.633666    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-trusted-ca-bundle\") pod \"47bd3e07-5f95-4203-a46f-23e5819cc710\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") "
Jul 06 20:52:09.633796 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.633698    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-rulefiles-1\") pod \"47bd3e07-5f95-4203-a46f-23e5819cc710\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") "
Jul 06 20:52:09.633796 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.633728    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-web-config\") pod \"47bd3e07-5f95-4203-a46f-23e5819cc710\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") "
Jul 06 20:52:09.633796 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.633772    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-configmap-metrics-client-ca\") pod \"47bd3e07-5f95-4203-a46f-23e5819cc710\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") "
Jul 06 20:52:09.633934 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.633799    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-db\") pod \"47bd3e07-5f95-4203-a46f-23e5819cc710\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") "
Jul 06 20:52:09.633934 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.633846    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-prometheus-k8s-tls\") pod \"47bd3e07-5f95-4203-a46f-23e5819cc710\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") "
Jul 06 20:52:09.633934 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.633870    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-grpc-tls\") pod \"47bd3e07-5f95-4203-a46f-23e5819cc710\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") "
Jul 06 20:52:09.633934 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.633903    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/47bd3e07-5f95-4203-a46f-23e5819cc710-tls-assets\") pod \"47bd3e07-5f95-4203-a46f-23e5819cc710\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") "
Jul 06 20:52:09.634115 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.633943    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-rulefiles-2\") pod \"47bd3e07-5f95-4203-a46f-23e5819cc710\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") "
Jul 06 20:52:09.634115 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.633973    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-config\") pod \"47bd3e07-5f95-4203-a46f-23e5819cc710\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") "
Jul 06 20:52:09.634115 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.633998    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"47bd3e07-5f95-4203-a46f-23e5819cc710\" (UID: \"47bd3e07-5f95-4203-a46f-23e5819cc710\") "
Jul 06 20:52:09.634395 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.634368    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "47bd3e07-5f95-4203-a46f-23e5819cc710" (UID: "47bd3e07-5f95-4203-a46f-23e5819cc710"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:52:09.635411 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.634968    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "47bd3e07-5f95-4203-a46f-23e5819cc710" (UID: "47bd3e07-5f95-4203-a46f-23e5819cc710"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:52:09.636143 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.635315    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "47bd3e07-5f95-4203-a46f-23e5819cc710" (UID: "47bd3e07-5f95-4203-a46f-23e5819cc710"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:52:09.637123 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.635728    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "47bd3e07-5f95-4203-a46f-23e5819cc710" (UID: "47bd3e07-5f95-4203-a46f-23e5819cc710"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:52:09.637123 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.635910    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "47bd3e07-5f95-4203-a46f-23e5819cc710" (UID: "47bd3e07-5f95-4203-a46f-23e5819cc710"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:52:09.637123 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.636736    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "47bd3e07-5f95-4203-a46f-23e5819cc710" (UID: "47bd3e07-5f95-4203-a46f-23e5819cc710"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:52:09.637123 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.637084    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "47bd3e07-5f95-4203-a46f-23e5819cc710" (UID: "47bd3e07-5f95-4203-a46f-23e5819cc710"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:52:09.637570 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.637498    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "47bd3e07-5f95-4203-a46f-23e5819cc710" (UID: "47bd3e07-5f95-4203-a46f-23e5819cc710"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:52:09.637570 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.637538    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "47bd3e07-5f95-4203-a46f-23e5819cc710" (UID: "47bd3e07-5f95-4203-a46f-23e5819cc710"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:52:09.638476 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.638451    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47bd3e07-5f95-4203-a46f-23e5819cc710-kube-api-access-8rz9s" (OuterVolumeSpecName: "kube-api-access-8rz9s") pod "47bd3e07-5f95-4203-a46f-23e5819cc710" (UID: "47bd3e07-5f95-4203-a46f-23e5819cc710"). InnerVolumeSpecName "kube-api-access-8rz9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:52:09.641328 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.641066    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "47bd3e07-5f95-4203-a46f-23e5819cc710" (UID: "47bd3e07-5f95-4203-a46f-23e5819cc710"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:52:09.641328 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.641155    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "47bd3e07-5f95-4203-a46f-23e5819cc710" (UID: "47bd3e07-5f95-4203-a46f-23e5819cc710"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:52:09.641328 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.641206    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "47bd3e07-5f95-4203-a46f-23e5819cc710" (UID: "47bd3e07-5f95-4203-a46f-23e5819cc710"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:52:09.641328 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.641259    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-config" (OuterVolumeSpecName: "config") pod "47bd3e07-5f95-4203-a46f-23e5819cc710" (UID: "47bd3e07-5f95-4203-a46f-23e5819cc710"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:52:09.641328 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.641299    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "47bd3e07-5f95-4203-a46f-23e5819cc710" (UID: "47bd3e07-5f95-4203-a46f-23e5819cc710"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:52:09.641823 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.641786    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47bd3e07-5f95-4203-a46f-23e5819cc710-config-out" (OuterVolumeSpecName: "config-out") pod "47bd3e07-5f95-4203-a46f-23e5819cc710" (UID: "47bd3e07-5f95-4203-a46f-23e5819cc710"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:52:09.641965 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.641945    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47bd3e07-5f95-4203-a46f-23e5819cc710-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "47bd3e07-5f95-4203-a46f-23e5819cc710" (UID: "47bd3e07-5f95-4203-a46f-23e5819cc710"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:52:09.642030 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.642004    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "47bd3e07-5f95-4203-a46f-23e5819cc710" (UID: "47bd3e07-5f95-4203-a46f-23e5819cc710"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:52:09.642229 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.642201    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "47bd3e07-5f95-4203-a46f-23e5819cc710" (UID: "47bd3e07-5f95-4203-a46f-23e5819cc710"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:52:09.650531 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.650503    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-web-config" (OuterVolumeSpecName: "web-config") pod "47bd3e07-5f95-4203-a46f-23e5819cc710" (UID: "47bd3e07-5f95-4203-a46f-23e5819cc710"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:52:09.735164 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.735132    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-db\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:52:09.735164 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.735160    2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:52:09.735164 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.735171    2559 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-grpc-tls\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:52:09.735389 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.735181    2559 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/47bd3e07-5f95-4203-a46f-23e5819cc710-tls-assets\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:52:09.735389 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.735189    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:52:09.735389 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.735197    2559 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:52:09.735389 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.735209    2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:52:09.735389 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.735218    2559 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:52:09.735389 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.735229    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rz9s\" (UniqueName: \"kubernetes.io/projected/47bd3e07-5f95-4203-a46f-23e5819cc710-kube-api-access-8rz9s\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:52:09.735389 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.735238    2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:52:09.735389 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.735246    2559 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-kube-rbac-proxy\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:52:09.735389 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.735254    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:52:09.735389 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.735262    2559 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:52:09.735389 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.735271    2559 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-secret-metrics-client-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:52:09.735389 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.735280    2559 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:52:09.735389 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.735287    2559 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/47bd3e07-5f95-4203-a46f-23e5819cc710-config-out\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:52:09.735389 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.735295    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:52:09.735389 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.735312    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:52:09.735389 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.735321    2559 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/47bd3e07-5f95-4203-a46f-23e5819cc710-web-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:52:09.735389 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:09.735328    2559 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47bd3e07-5f95-4203-a46f-23e5819cc710-configmap-metrics-client-ca\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:52:10.558180 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.558148    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47bd3e07-5f95-4203-a46f-23e5819cc710","Type":"ContainerDied","Data":"01883a606c73b07d1a8c50a28ea8c82ad5f1573f3a4a954343bd1c556a87754f"}
Jul 06 20:52:10.558180 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.558176    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.558699 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.558232    2559 scope.go:117] "RemoveContainer" containerID="75dbf22d340e6be6220b2292149c7937d4579197149c3ac2cdabcbdf81ddb4c6"
Jul 06 20:52:10.566092 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.566074    2559 scope.go:117] "RemoveContainer" containerID="838593a0a3cdc964f8d184b2deac8aa20a14d23873be3ee0b9cfb72e003cb781"
Jul 06 20:52:10.573115 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.573089    2559 scope.go:117] "RemoveContainer" containerID="38b78d7a3456af84a585e6aae104893d4590ac75284cd8e71d45ec903d70b382"
Jul 06 20:52:10.579406 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.579393    2559 scope.go:117] "RemoveContainer" containerID="8793af8b711b9956e838fd8694c991cc7a056b44ae9e4bb1dc19651f6df9a09e"
Jul 06 20:52:10.582424 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.582405    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:52:10.586139 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.586124    2559 scope.go:117] "RemoveContainer" containerID="7078160252b47bfcdb2b059b86f164cd624a4e3534ddb197b54d6a9faf055664"
Jul 06 20:52:10.588531 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.588515    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:52:10.593536 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.593520    2559 scope.go:117] "RemoveContainer" containerID="f239b9f6ca4ac9b6c581b2af849b24008480b8f664ef996f4801a994979bd7cb"
Jul 06 20:52:10.600294 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.600280    2559 scope.go:117] "RemoveContainer" containerID="d85896a766d676947ad3e1512fa22c5ac9431aec743d28ec0d6b28885ed1be23"
Jul 06 20:52:10.625179 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.625158    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:52:10.625536 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.625522    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="kube-rbac-proxy-thanos"
Jul 06 20:52:10.625584 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.625539    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="kube-rbac-proxy-thanos"
Jul 06 20:52:10.625584 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.625557    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="thanos-sidecar"
Jul 06 20:52:10.625584 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.625563    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="thanos-sidecar"
Jul 06 20:52:10.625584 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.625570    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="kube-rbac-proxy"
Jul 06 20:52:10.625584 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.625576    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="kube-rbac-proxy"
Jul 06 20:52:10.625738 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.625589    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="kube-rbac-proxy-web"
Jul 06 20:52:10.625738 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.625595    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="kube-rbac-proxy-web"
Jul 06 20:52:10.625738 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.625601    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="config-reloader"
Jul 06 20:52:10.625738 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.625620    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="config-reloader"
Jul 06 20:52:10.625738 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.625633    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="init-config-reloader"
Jul 06 20:52:10.625738 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.625641    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="init-config-reloader"
Jul 06 20:52:10.625738 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.625649    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="prometheus"
Jul 06 20:52:10.625738 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.625654    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="prometheus"
Jul 06 20:52:10.625738 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.625702    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="kube-rbac-proxy"
Jul 06 20:52:10.625738 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.625710    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="prometheus"
Jul 06 20:52:10.625738 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.625716    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="thanos-sidecar"
Jul 06 20:52:10.625738 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.625722    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="kube-rbac-proxy-web"
Jul 06 20:52:10.625738 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.625728    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="config-reloader"
Jul 06 20:52:10.625738 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.625733    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" containerName="kube-rbac-proxy-thanos"
Jul 06 20:52:10.629529 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.629512    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.632206 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.632188    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 06 20:52:10.633219 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.633198    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x6w6j\""
Jul 06 20:52:10.633303 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.633265    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 06 20:52:10.633583 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.633570    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 06 20:52:10.633665 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.633573    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 06 20:52:10.633665 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.633644    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 06 20:52:10.633763 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.633725    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 06 20:52:10.633861 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.633842    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bj4p8s0b8opik\""
Jul 06 20:52:10.634553 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.634537    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 06 20:52:10.635558 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.635542    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 06 20:52:10.636263 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.636246    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 06 20:52:10.636342 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.636275    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 06 20:52:10.636462 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.636447    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 06 20:52:10.636803 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.636766    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 06 20:52:10.639391 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.639370    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 06 20:52:10.642488 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.642466    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 06 20:52:10.651895 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.651876    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:52:10.744378 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.744354    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.744508 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.744385    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.744508 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.744404    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2cce5eb-5268-4340-af2f-5e3e3e383690-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.744508 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.744422    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d2cce5eb-5268-4340-af2f-5e3e3e383690-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.744508 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.744466    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d2cce5eb-5268-4340-af2f-5e3e3e383690-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.744817 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.744516    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d2cce5eb-5268-4340-af2f-5e3e3e383690-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.744817 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.744551    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.744817 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.744599    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2cce5eb-5268-4340-af2f-5e3e3e383690-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.744817 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.744652    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d2cce5eb-5268-4340-af2f-5e3e3e383690-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.744817 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.744679    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2cce5eb-5268-4340-af2f-5e3e3e383690-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.744817 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.744706    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.744817 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.744735    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.744817 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.744769    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2cce5eb-5268-4340-af2f-5e3e3e383690-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.745126 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.744824    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.745126 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.744853    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-config\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.745126 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.744903    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-web-config\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.745126 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.744943    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d2cce5eb-5268-4340-af2f-5e3e3e383690-config-out\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.745126 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.744981    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndgrp\" (UniqueName: \"kubernetes.io/projected/d2cce5eb-5268-4340-af2f-5e3e3e383690-kube-api-access-ndgrp\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.745126 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.744998    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.745126 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.745015    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d2cce5eb-5268-4340-af2f-5e3e3e383690-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.846229 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.846167    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndgrp\" (UniqueName: \"kubernetes.io/projected/d2cce5eb-5268-4340-af2f-5e3e3e383690-kube-api-access-ndgrp\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.846229 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.846197    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.846229 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.846214    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d2cce5eb-5268-4340-af2f-5e3e3e383690-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.846452 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.846235    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.846452 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.846423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.846553 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.846458    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2cce5eb-5268-4340-af2f-5e3e3e383690-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.846553 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.846490    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d2cce5eb-5268-4340-af2f-5e3e3e383690-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.846553 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.846517    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d2cce5eb-5268-4340-af2f-5e3e3e383690-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.846731 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.846558    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d2cce5eb-5268-4340-af2f-5e3e3e383690-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.846731 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.846597    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.846731 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.846667    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2cce5eb-5268-4340-af2f-5e3e3e383690-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.846731 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.846692    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d2cce5eb-5268-4340-af2f-5e3e3e383690-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.846731 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.846717    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2cce5eb-5268-4340-af2f-5e3e3e383690-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.846998 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.846756    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.846998 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.846803    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.846998 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.846845    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2cce5eb-5268-4340-af2f-5e3e3e383690-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.846998 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.846890    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.846998 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.846916    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-config\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.846998 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.846942    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-web-config\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.846998 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.846971    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d2cce5eb-5268-4340-af2f-5e3e3e383690-config-out\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.847345 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.847321    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2cce5eb-5268-4340-af2f-5e3e3e383690-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.848801 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.848067    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d2cce5eb-5268-4340-af2f-5e3e3e383690-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.848801 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.848661    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2cce5eb-5268-4340-af2f-5e3e3e383690-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.848960 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.848894    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d2cce5eb-5268-4340-af2f-5e3e3e383690-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.851322 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.849213    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.851322 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.849385    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d2cce5eb-5268-4340-af2f-5e3e3e383690-config-out\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.851322 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.849559    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d2cce5eb-5268-4340-af2f-5e3e3e383690-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.851322 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.849698    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d2cce5eb-5268-4340-af2f-5e3e3e383690-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.851322 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.850025    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.851322 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.850277    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.851322 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.850468    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2cce5eb-5268-4340-af2f-5e3e3e383690-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.851322 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.850828    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.851322 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.851113    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2cce5eb-5268-4340-af2f-5e3e3e383690-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.851849 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.851656    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.852375 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.852329    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.852671 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.852648    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d2cce5eb-5268-4340-af2f-5e3e3e383690-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.852761 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.852656    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-config\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.852953 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.852934    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.853486 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.853468    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d2cce5eb-5268-4340-af2f-5e3e3e383690-web-config\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.855405 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.855388    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndgrp\" (UniqueName: \"kubernetes.io/projected/d2cce5eb-5268-4340-af2f-5e3e3e383690-kube-api-access-ndgrp\") pod \"prometheus-k8s-0\" (UID: \"d2cce5eb-5268-4340-af2f-5e3e3e383690\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:10.940590 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:10.940563    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:52:11.075642 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:11.075602    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:52:11.078025 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:52:11.078003    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2cce5eb_5268_4340_af2f_5e3e3e383690.slice/crio-1efdba6ec1b5b3bd0afff199ee72c69087935105f80476ac4ba7ec4e7e9cac00 WatchSource:0}: Error finding container 1efdba6ec1b5b3bd0afff199ee72c69087935105f80476ac4ba7ec4e7e9cac00: Status 404 returned error can't find the container with id 1efdba6ec1b5b3bd0afff199ee72c69087935105f80476ac4ba7ec4e7e9cac00
Jul 06 20:52:11.563646 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:11.563601    2559 generic.go:356] "Generic (PLEG): container finished" podID="d2cce5eb-5268-4340-af2f-5e3e3e383690" containerID="fb0c2db33bbdb200b88a8411e3b75c35f5e56a0e5d0dadee9611971026f196fc" exitCode=0
Jul 06 20:52:11.564046 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:11.563638    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2cce5eb-5268-4340-af2f-5e3e3e383690","Type":"ContainerDied","Data":"fb0c2db33bbdb200b88a8411e3b75c35f5e56a0e5d0dadee9611971026f196fc"}
Jul 06 20:52:11.564046 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:11.563719    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2cce5eb-5268-4340-af2f-5e3e3e383690","Type":"ContainerStarted","Data":"1efdba6ec1b5b3bd0afff199ee72c69087935105f80476ac4ba7ec4e7e9cac00"}
Jul 06 20:52:11.660038 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:11.660008    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47bd3e07-5f95-4203-a46f-23e5819cc710" path="/var/lib/kubelet/pods/47bd3e07-5f95-4203-a46f-23e5819cc710/volumes"
Jul 06 20:52:12.570889 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:12.570855    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2cce5eb-5268-4340-af2f-5e3e3e383690","Type":"ContainerStarted","Data":"b1f6f1caa5dc03416062da9dd009ac0b7e3fd02c1df61fe1592e22a156d11166"}
Jul 06 20:52:12.571290 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:12.570938    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2cce5eb-5268-4340-af2f-5e3e3e383690","Type":"ContainerStarted","Data":"a78034cb7fae99e468866752fbba8826c2f1a7496b1dab138e8b71d4954bd32d"}
Jul 06 20:52:12.571290 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:12.570977    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2cce5eb-5268-4340-af2f-5e3e3e383690","Type":"ContainerStarted","Data":"9e9f7e1e00fd80defa6dd335df94f1d746838e30ef72b683bf0da8da498c54d2"}
Jul 06 20:52:12.571290 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:12.571021    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2cce5eb-5268-4340-af2f-5e3e3e383690","Type":"ContainerStarted","Data":"9675a0c318e4cda9ee9119a5a8b98fdb151af4e8aec095538a948d85231ba8b0"}
Jul 06 20:52:12.571290 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:12.571057    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2cce5eb-5268-4340-af2f-5e3e3e383690","Type":"ContainerStarted","Data":"0affded904f432404cbeec9f1f48f18a0ef7570b455dd3d36e5c835b0925b3c9"}
Jul 06 20:52:12.571290 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:12.571090    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2cce5eb-5268-4340-af2f-5e3e3e383690","Type":"ContainerStarted","Data":"12fa25a3011a904a5ef4386b5876342d8d52c64c7936432ac3a35eca323be9db"}
Jul 06 20:52:12.609925 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:12.609883    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.587982521 podStartE2EDuration="2.609847788s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.865267ms" isStatefulPod=false podCreationTimestamp="2026-07-06 20:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:52:12.608518517 +0000 UTC m=+181.443405881" watchObservedRunningTime="2026-07-06 20:52:12.609847788 +0000 UTC m=+181.444735151"
Jul 06 20:52:15.940929 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:52:15.940897    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:53:10.941065 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:53:10.941030    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:53:10.955939 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:53:10.955915    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:53:11.817428 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:53:11.817387    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:54:11.541851 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:54:11.541826    2559 kubelet.go:1648] "Image garbage collection succeeded"
Jul 06 20:57:09.394347 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:09.394319    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mrth2"]
Jul 06 20:57:09.396678 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:09.396660    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mrth2"
Jul 06 20:57:09.408271 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:09.408248    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 06 20:57:09.409203 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:09.409182    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 06 20:57:09.409314 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:09.409246    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-t69b6\""
Jul 06 20:57:09.410472 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:09.410450    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mrth2"]
Jul 06 20:57:09.452105 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:09.452073    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k59ts\" (UniqueName: \"kubernetes.io/projected/b06ca78c-7292-4002-8a1a-0ef37a9c96ac-kube-api-access-k59ts\") pod \"cert-manager-cainjector-559cdd8c87-mrth2\" (UID: \"b06ca78c-7292-4002-8a1a-0ef37a9c96ac\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mrth2"
Jul 06 20:57:09.452219 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:09.452120    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b06ca78c-7292-4002-8a1a-0ef37a9c96ac-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mrth2\" (UID: \"b06ca78c-7292-4002-8a1a-0ef37a9c96ac\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mrth2"
Jul 06 20:57:09.552766 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:09.552735    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k59ts\" (UniqueName: \"kubernetes.io/projected/b06ca78c-7292-4002-8a1a-0ef37a9c96ac-kube-api-access-k59ts\") pod \"cert-manager-cainjector-559cdd8c87-mrth2\" (UID: \"b06ca78c-7292-4002-8a1a-0ef37a9c96ac\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mrth2"
Jul 06 20:57:09.552907 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:09.552777    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b06ca78c-7292-4002-8a1a-0ef37a9c96ac-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mrth2\" (UID: \"b06ca78c-7292-4002-8a1a-0ef37a9c96ac\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mrth2"
Jul 06 20:57:09.569701 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:09.569675    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k59ts\" (UniqueName: \"kubernetes.io/projected/b06ca78c-7292-4002-8a1a-0ef37a9c96ac-kube-api-access-k59ts\") pod \"cert-manager-cainjector-559cdd8c87-mrth2\" (UID: \"b06ca78c-7292-4002-8a1a-0ef37a9c96ac\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mrth2"
Jul 06 20:57:09.573716 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:09.573697    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b06ca78c-7292-4002-8a1a-0ef37a9c96ac-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mrth2\" (UID: \"b06ca78c-7292-4002-8a1a-0ef37a9c96ac\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mrth2"
Jul 06 20:57:09.719115 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:09.719032    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mrth2"
Jul 06 20:57:09.851341 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:09.851195    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mrth2"]
Jul 06 20:57:09.854130 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:57:09.854103    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb06ca78c_7292_4002_8a1a_0ef37a9c96ac.slice/crio-596cad2b5f3c841716a3d7505d84c4f82b2ef7db413ca2184745684044bdb254 WatchSource:0}: Error finding container 596cad2b5f3c841716a3d7505d84c4f82b2ef7db413ca2184745684044bdb254: Status 404 returned error can't find the container with id 596cad2b5f3c841716a3d7505d84c4f82b2ef7db413ca2184745684044bdb254
Jul 06 20:57:09.856018 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:09.855998    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 20:57:10.742043 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:10.741997    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mrth2" event={"ID":"b06ca78c-7292-4002-8a1a-0ef37a9c96ac","Type":"ContainerStarted","Data":"596cad2b5f3c841716a3d7505d84c4f82b2ef7db413ca2184745684044bdb254"}
Jul 06 20:57:13.757861 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:13.757829    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mrth2" event={"ID":"b06ca78c-7292-4002-8a1a-0ef37a9c96ac","Type":"ContainerStarted","Data":"a14622ad4c1a95a6542eb5cbb4f2a8515310cdeae172488b63ea13aaf14f2c7f"}
Jul 06 20:57:13.775762 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:13.775721    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mrth2" podStartSLOduration=1.518993281 podStartE2EDuration="4.775704981s" totalImagesPullingTime="3.2567117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:57:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:57:13.774278721 +0000 UTC m=+482.609166087" watchObservedRunningTime="2026-07-06 20:57:13.775704981 +0000 UTC m=+482.610592348"
Jul 06 20:57:16.133366 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:16.133335    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gs9f9"]
Jul 06 20:57:16.136700 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:16.136683    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-gs9f9"
Jul 06 20:57:16.139034 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:16.139011    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-hrt27\""
Jul 06 20:57:16.147592 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:16.147573    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gs9f9"]
Jul 06 20:57:16.211630 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:16.211589    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n68hk\" (UniqueName: \"kubernetes.io/projected/09764595-2dcf-443c-8fdd-98c51de914a8-kube-api-access-n68hk\") pod \"cert-manager-55b4d954d5-gs9f9\" (UID: \"09764595-2dcf-443c-8fdd-98c51de914a8\") " pod="cert-manager/cert-manager-55b4d954d5-gs9f9"
Jul 06 20:57:16.211801 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:16.211666    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09764595-2dcf-443c-8fdd-98c51de914a8-bound-sa-token\") pod \"cert-manager-55b4d954d5-gs9f9\" (UID: \"09764595-2dcf-443c-8fdd-98c51de914a8\") " pod="cert-manager/cert-manager-55b4d954d5-gs9f9"
Jul 06 20:57:16.312851 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:16.312815    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n68hk\" (UniqueName: \"kubernetes.io/projected/09764595-2dcf-443c-8fdd-98c51de914a8-kube-api-access-n68hk\") pod \"cert-manager-55b4d954d5-gs9f9\" (UID: \"09764595-2dcf-443c-8fdd-98c51de914a8\") " pod="cert-manager/cert-manager-55b4d954d5-gs9f9"
Jul 06 20:57:16.312994 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:16.312862    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09764595-2dcf-443c-8fdd-98c51de914a8-bound-sa-token\") pod \"cert-manager-55b4d954d5-gs9f9\" (UID: \"09764595-2dcf-443c-8fdd-98c51de914a8\") " pod="cert-manager/cert-manager-55b4d954d5-gs9f9"
Jul 06 20:57:16.323520 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:16.323490    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09764595-2dcf-443c-8fdd-98c51de914a8-bound-sa-token\") pod \"cert-manager-55b4d954d5-gs9f9\" (UID: \"09764595-2dcf-443c-8fdd-98c51de914a8\") " pod="cert-manager/cert-manager-55b4d954d5-gs9f9"
Jul 06 20:57:16.323650 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:16.323631    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n68hk\" (UniqueName: \"kubernetes.io/projected/09764595-2dcf-443c-8fdd-98c51de914a8-kube-api-access-n68hk\") pod \"cert-manager-55b4d954d5-gs9f9\" (UID: \"09764595-2dcf-443c-8fdd-98c51de914a8\") " pod="cert-manager/cert-manager-55b4d954d5-gs9f9"
Jul 06 20:57:16.445257 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:16.445185    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-gs9f9"
Jul 06 20:57:16.571058 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:16.571028    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gs9f9"]
Jul 06 20:57:16.574296 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:57:16.574269    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09764595_2dcf_443c_8fdd_98c51de914a8.slice/crio-984cae91f6f5a52afa8c82fb10704dd9c8fe41c5ee6a2a8bc9a874d487998911 WatchSource:0}: Error finding container 984cae91f6f5a52afa8c82fb10704dd9c8fe41c5ee6a2a8bc9a874d487998911: Status 404 returned error can't find the container with id 984cae91f6f5a52afa8c82fb10704dd9c8fe41c5ee6a2a8bc9a874d487998911
Jul 06 20:57:16.773385 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:16.773299    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-gs9f9" event={"ID":"09764595-2dcf-443c-8fdd-98c51de914a8","Type":"ContainerStarted","Data":"1d22557b149b24377423a67c5a85c9445d07cb6d708003ea0e53772be9283957"}
Jul 06 20:57:16.773385 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:16.773341    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-gs9f9" event={"ID":"09764595-2dcf-443c-8fdd-98c51de914a8","Type":"ContainerStarted","Data":"984cae91f6f5a52afa8c82fb10704dd9c8fe41c5ee6a2a8bc9a874d487998911"}
Jul 06 20:57:16.790018 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:16.789975    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-gs9f9" podStartSLOduration=0.78996062 podStartE2EDuration="789.96062ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:57:16.788695317 +0000 UTC m=+485.623582695" watchObservedRunningTime="2026-07-06 20:57:16.78996062 +0000 UTC m=+485.624847985"
Jul 06 20:57:50.107816 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.107782    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7694784d57-k44g7"]
Jul 06 20:57:50.110619 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.110579    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7694784d57-k44g7"
Jul 06 20:57:50.112847 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.112818    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 06 20:57:50.113472 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.113455    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:57:50.113571 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.113476    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 06 20:57:50.113674 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.113659    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 06 20:57:50.113787 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.113764    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 06 20:57:50.114110 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.114097    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-b66t2\""
Jul 06 20:57:50.125207 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.125184    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7694784d57-k44g7"]
Jul 06 20:57:50.192828 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.192781    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9fc927d2-be1f-43ec-9b2b-5ec63c5d0859-metrics-cert\") pod \"lws-controller-manager-7694784d57-k44g7\" (UID: \"9fc927d2-be1f-43ec-9b2b-5ec63c5d0859\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-k44g7"
Jul 06 20:57:50.193030 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.192840    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fc927d2-be1f-43ec-9b2b-5ec63c5d0859-cert\") pod \"lws-controller-manager-7694784d57-k44g7\" (UID: \"9fc927d2-be1f-43ec-9b2b-5ec63c5d0859\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-k44g7"
Jul 06 20:57:50.193030 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.192914    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9fc927d2-be1f-43ec-9b2b-5ec63c5d0859-manager-config\") pod \"lws-controller-manager-7694784d57-k44g7\" (UID: \"9fc927d2-be1f-43ec-9b2b-5ec63c5d0859\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-k44g7"
Jul 06 20:57:50.193030 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.192978    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pft2z\" (UniqueName: \"kubernetes.io/projected/9fc927d2-be1f-43ec-9b2b-5ec63c5d0859-kube-api-access-pft2z\") pod \"lws-controller-manager-7694784d57-k44g7\" (UID: \"9fc927d2-be1f-43ec-9b2b-5ec63c5d0859\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-k44g7"
Jul 06 20:57:50.293866 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.293829    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pft2z\" (UniqueName: \"kubernetes.io/projected/9fc927d2-be1f-43ec-9b2b-5ec63c5d0859-kube-api-access-pft2z\") pod \"lws-controller-manager-7694784d57-k44g7\" (UID: \"9fc927d2-be1f-43ec-9b2b-5ec63c5d0859\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-k44g7"
Jul 06 20:57:50.294041 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.293888    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9fc927d2-be1f-43ec-9b2b-5ec63c5d0859-metrics-cert\") pod \"lws-controller-manager-7694784d57-k44g7\" (UID: \"9fc927d2-be1f-43ec-9b2b-5ec63c5d0859\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-k44g7"
Jul 06 20:57:50.294041 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.293923    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fc927d2-be1f-43ec-9b2b-5ec63c5d0859-cert\") pod \"lws-controller-manager-7694784d57-k44g7\" (UID: \"9fc927d2-be1f-43ec-9b2b-5ec63c5d0859\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-k44g7"
Jul 06 20:57:50.294041 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.293947    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9fc927d2-be1f-43ec-9b2b-5ec63c5d0859-manager-config\") pod \"lws-controller-manager-7694784d57-k44g7\" (UID: \"9fc927d2-be1f-43ec-9b2b-5ec63c5d0859\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-k44g7"
Jul 06 20:57:50.294795 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.294769    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9fc927d2-be1f-43ec-9b2b-5ec63c5d0859-manager-config\") pod \"lws-controller-manager-7694784d57-k44g7\" (UID: \"9fc927d2-be1f-43ec-9b2b-5ec63c5d0859\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-k44g7"
Jul 06 20:57:50.296358 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.296336    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fc927d2-be1f-43ec-9b2b-5ec63c5d0859-cert\") pod \"lws-controller-manager-7694784d57-k44g7\" (UID: \"9fc927d2-be1f-43ec-9b2b-5ec63c5d0859\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-k44g7"
Jul 06 20:57:50.296491 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.296471    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9fc927d2-be1f-43ec-9b2b-5ec63c5d0859-metrics-cert\") pod \"lws-controller-manager-7694784d57-k44g7\" (UID: \"9fc927d2-be1f-43ec-9b2b-5ec63c5d0859\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-k44g7"
Jul 06 20:57:50.302704 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.302684    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pft2z\" (UniqueName: \"kubernetes.io/projected/9fc927d2-be1f-43ec-9b2b-5ec63c5d0859-kube-api-access-pft2z\") pod \"lws-controller-manager-7694784d57-k44g7\" (UID: \"9fc927d2-be1f-43ec-9b2b-5ec63c5d0859\") " pod="openshift-lws-operator/lws-controller-manager-7694784d57-k44g7"
Jul 06 20:57:50.420828 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.420747    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7694784d57-k44g7"
Jul 06 20:57:50.555194 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.555159    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7694784d57-k44g7"]
Jul 06 20:57:50.559319 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:57:50.559289    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc927d2_be1f_43ec_9b2b_5ec63c5d0859.slice/crio-23ec008374ed76abc51c8ac613d3c6fa9ec00e62f69c8183710eb5abbbad4a4b WatchSource:0}: Error finding container 23ec008374ed76abc51c8ac613d3c6fa9ec00e62f69c8183710eb5abbbad4a4b: Status 404 returned error can't find the container with id 23ec008374ed76abc51c8ac613d3c6fa9ec00e62f69c8183710eb5abbbad4a4b
Jul 06 20:57:50.911603 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:50.911547    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7694784d57-k44g7" event={"ID":"9fc927d2-be1f-43ec-9b2b-5ec63c5d0859","Type":"ContainerStarted","Data":"23ec008374ed76abc51c8ac613d3c6fa9ec00e62f69c8183710eb5abbbad4a4b"}
Jul 06 20:57:53.927623 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:53.927574    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7694784d57-k44g7" event={"ID":"9fc927d2-be1f-43ec-9b2b-5ec63c5d0859","Type":"ContainerStarted","Data":"a78344be7c18f05c802c14953bbab3f2f576294cb1373c872be187b77300914c"}
Jul 06 20:57:53.928040 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:53.927709    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7694784d57-k44g7"
Jul 06 20:57:53.947517 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:57:53.947466    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7694784d57-k44g7" podStartSLOduration=1.502585407 podStartE2EDuration="3.947451091s" totalImagesPullingTime="2.444865684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:57:53.946788757 +0000 UTC m=+522.781676159" watchObservedRunningTime="2026-07-06 20:57:53.947451091 +0000 UTC m=+522.782338456"
Jul 06 20:58:04.934403 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:58:04.934324    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7694784d57-k44g7"
Jul 06 20:58:15.885244 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:58:15.885205    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-w2wzk"]
Jul 06 20:58:15.891524 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:58:15.891502    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-w2wzk"
Jul 06 20:58:15.893867 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:58:15.893843    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 06 20:58:15.893996 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:58:15.893882    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 06 20:58:15.894529 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:58:15.894510    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-m62dz\""
Jul 06 20:58:15.898597 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:58:15.898536    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-w2wzk"]
Jul 06 20:58:16.024674 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:58:16.022488    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s5n8\" (UniqueName: \"kubernetes.io/projected/c2582e4d-80db-4433-9393-eee2761f443c-kube-api-access-4s5n8\") pod \"limitador-operator-controller-manager-6f9f468797-w2wzk\" (UID: \"c2582e4d-80db-4433-9393-eee2761f443c\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-w2wzk"
Jul 06 20:58:16.123873 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:58:16.123833    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s5n8\" (UniqueName: \"kubernetes.io/projected/c2582e4d-80db-4433-9393-eee2761f443c-kube-api-access-4s5n8\") pod \"limitador-operator-controller-manager-6f9f468797-w2wzk\" (UID: \"c2582e4d-80db-4433-9393-eee2761f443c\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-w2wzk"
Jul 06 20:58:16.135639 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:58:16.135567    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s5n8\" (UniqueName: \"kubernetes.io/projected/c2582e4d-80db-4433-9393-eee2761f443c-kube-api-access-4s5n8\") pod \"limitador-operator-controller-manager-6f9f468797-w2wzk\" (UID: \"c2582e4d-80db-4433-9393-eee2761f443c\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-w2wzk"
Jul 06 20:58:16.202408 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:58:16.202372    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-w2wzk"
Jul 06 20:58:16.334886 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:58:16.334861    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-w2wzk"]
Jul 06 20:58:16.337441 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:58:16.337399    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2582e4d_80db_4433_9393_eee2761f443c.slice/crio-97416372e9a8f555b12471856e2155d99a9cd7d367a04e25cc648d80a58a3447 WatchSource:0}: Error finding container 97416372e9a8f555b12471856e2155d99a9cd7d367a04e25cc648d80a58a3447: Status 404 returned error can't find the container with id 97416372e9a8f555b12471856e2155d99a9cd7d367a04e25cc648d80a58a3447
Jul 06 20:58:17.028786 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:58:17.028748    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-w2wzk" event={"ID":"c2582e4d-80db-4433-9393-eee2761f443c","Type":"ContainerStarted","Data":"97416372e9a8f555b12471856e2155d99a9cd7d367a04e25cc648d80a58a3447"}
Jul 06 20:58:18.036165 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:58:18.036125    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-w2wzk" event={"ID":"c2582e4d-80db-4433-9393-eee2761f443c","Type":"ContainerStarted","Data":"91445a33cb0d2ecf50f6568cc14beef5fe7134171ad288b38dc5adbff887c88e"}
Jul 06 20:58:18.036535 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:58:18.036198    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-w2wzk"
Jul 06 20:58:18.057872 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:58:18.057824    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-w2wzk" podStartSLOduration=1.7848665989999999 podStartE2EDuration="3.057807259s" totalImagesPullingTime="1.27294066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:58:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:58:18.057529351 +0000 UTC m=+546.892416716" watchObservedRunningTime="2026-07-06 20:58:18.057807259 +0000 UTC m=+546.892694624"
Jul 06 20:58:29.042828 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:58:29.042795    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-w2wzk"
Jul 06 20:59:06.091353 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:06.091319    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-nqg8v"]
Jul 06 20:59:06.094555 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:06.094536    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-69574b596d-nqg8v"
Jul 06 20:59:06.097253 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:06.097233    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 06 20:59:06.097623 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:06.097591    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9zp7s\""
Jul 06 20:59:06.113994 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:06.113970    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-nqg8v"]
Jul 06 20:59:06.193446 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:06.193412    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-nqg8v"]
Jul 06 20:59:06.252894 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:06.252851    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp5fx\" (UniqueName: \"kubernetes.io/projected/dd7b6366-25ec-445a-beed-9ba9205afd53-kube-api-access-cp5fx\") pod \"limitador-limitador-69574b596d-nqg8v\" (UID: \"dd7b6366-25ec-445a-beed-9ba9205afd53\") " pod="kuadrant-system/limitador-limitador-69574b596d-nqg8v"
Jul 06 20:59:06.253072 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:06.252952    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dd7b6366-25ec-445a-beed-9ba9205afd53-config-file\") pod \"limitador-limitador-69574b596d-nqg8v\" (UID: \"dd7b6366-25ec-445a-beed-9ba9205afd53\") " pod="kuadrant-system/limitador-limitador-69574b596d-nqg8v"
Jul 06 20:59:06.354081 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:06.353999    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dd7b6366-25ec-445a-beed-9ba9205afd53-config-file\") pod \"limitador-limitador-69574b596d-nqg8v\" (UID: \"dd7b6366-25ec-445a-beed-9ba9205afd53\") " pod="kuadrant-system/limitador-limitador-69574b596d-nqg8v"
Jul 06 20:59:06.354081 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:06.354077    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp5fx\" (UniqueName: \"kubernetes.io/projected/dd7b6366-25ec-445a-beed-9ba9205afd53-kube-api-access-cp5fx\") pod \"limitador-limitador-69574b596d-nqg8v\" (UID: \"dd7b6366-25ec-445a-beed-9ba9205afd53\") " pod="kuadrant-system/limitador-limitador-69574b596d-nqg8v"
Jul 06 20:59:06.354736 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:06.354717    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dd7b6366-25ec-445a-beed-9ba9205afd53-config-file\") pod \"limitador-limitador-69574b596d-nqg8v\" (UID: \"dd7b6366-25ec-445a-beed-9ba9205afd53\") " pod="kuadrant-system/limitador-limitador-69574b596d-nqg8v"
Jul 06 20:59:06.363756 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:06.363734    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp5fx\" (UniqueName: \"kubernetes.io/projected/dd7b6366-25ec-445a-beed-9ba9205afd53-kube-api-access-cp5fx\") pod \"limitador-limitador-69574b596d-nqg8v\" (UID: \"dd7b6366-25ec-445a-beed-9ba9205afd53\") " pod="kuadrant-system/limitador-limitador-69574b596d-nqg8v"
Jul 06 20:59:06.404216 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:06.404175    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-69574b596d-nqg8v"
Jul 06 20:59:06.544244 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:06.544218    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-nqg8v"]
Jul 06 20:59:06.546956 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:59:06.546929    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7b6366_25ec_445a_beed_9ba9205afd53.slice/crio-eb55464ec0e3be85e8371e4a7242d7f18620afc424352ac6910f81e9d29a7c48 WatchSource:0}: Error finding container eb55464ec0e3be85e8371e4a7242d7f18620afc424352ac6910f81e9d29a7c48: Status 404 returned error can't find the container with id eb55464ec0e3be85e8371e4a7242d7f18620afc424352ac6910f81e9d29a7c48
Jul 06 20:59:06.794123 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:06.794094    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8df697c9f-shs9j"]
Jul 06 20:59:06.798198 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:06.798181    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-shs9j"
Jul 06 20:59:06.800550 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:06.800527    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4n9bd\""
Jul 06 20:59:06.806850 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:06.806829    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-shs9j"]
Jul 06 20:59:06.959943 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:06.959910    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kngpg\" (UniqueName: \"kubernetes.io/projected/f613c345-9ef3-4d95-97f8-68257a9805c6-kube-api-access-kngpg\") pod \"authorino-8df697c9f-shs9j\" (UID: \"f613c345-9ef3-4d95-97f8-68257a9805c6\") " pod="kuadrant-system/authorino-8df697c9f-shs9j"
Jul 06 20:59:07.060832 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:07.060745    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kngpg\" (UniqueName: \"kubernetes.io/projected/f613c345-9ef3-4d95-97f8-68257a9805c6-kube-api-access-kngpg\") pod \"authorino-8df697c9f-shs9j\" (UID: \"f613c345-9ef3-4d95-97f8-68257a9805c6\") " pod="kuadrant-system/authorino-8df697c9f-shs9j"
Jul 06 20:59:07.076170 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:07.076144    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kngpg\" (UniqueName: \"kubernetes.io/projected/f613c345-9ef3-4d95-97f8-68257a9805c6-kube-api-access-kngpg\") pod \"authorino-8df697c9f-shs9j\" (UID: \"f613c345-9ef3-4d95-97f8-68257a9805c6\") " pod="kuadrant-system/authorino-8df697c9f-shs9j"
Jul 06 20:59:07.108662 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:07.108629    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-shs9j"
Jul 06 20:59:07.262486 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:07.262448    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-69574b596d-nqg8v" event={"ID":"dd7b6366-25ec-445a-beed-9ba9205afd53","Type":"ContainerStarted","Data":"eb55464ec0e3be85e8371e4a7242d7f18620afc424352ac6910f81e9d29a7c48"}
Jul 06 20:59:07.274543 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:07.274494    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-shs9j"]
Jul 06 20:59:07.278541 ip-10-0-134-202 kubenswrapper[2559]: W0706 20:59:07.278509    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf613c345_9ef3_4d95_97f8_68257a9805c6.slice/crio-7fa9339995cef49cb42b66ac26e0128aab63bfc51a55b50742adea0a9e6e7392 WatchSource:0}: Error finding container 7fa9339995cef49cb42b66ac26e0128aab63bfc51a55b50742adea0a9e6e7392: Status 404 returned error can't find the container with id 7fa9339995cef49cb42b66ac26e0128aab63bfc51a55b50742adea0a9e6e7392
Jul 06 20:59:08.270790 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:08.270728    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-shs9j" event={"ID":"f613c345-9ef3-4d95-97f8-68257a9805c6","Type":"ContainerStarted","Data":"7fa9339995cef49cb42b66ac26e0128aab63bfc51a55b50742adea0a9e6e7392"}
Jul 06 20:59:12.293098 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:12.293061    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-69574b596d-nqg8v" event={"ID":"dd7b6366-25ec-445a-beed-9ba9205afd53","Type":"ContainerStarted","Data":"5a419f6553a1905501507da89ab314fe6985f7cdc0b775045c1d1dc1d7318984"}
Jul 06 20:59:12.293526 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:12.293124    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-69574b596d-nqg8v"
Jul 06 20:59:12.294430 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:12.294409    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-shs9j" event={"ID":"f613c345-9ef3-4d95-97f8-68257a9805c6","Type":"ContainerStarted","Data":"e94ce9848345f4a2d5adeae6908a2094f64ea0e43da981c7eb691eb4156b29eb"}
Jul 06 20:59:12.312965 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:12.312781    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-69574b596d-nqg8v" podStartSLOduration=1.498206068 podStartE2EDuration="6.312765427s" totalImagesPullingTime="4.814559359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:12.312405697 +0000 UTC m=+601.147293100" watchObservedRunningTime="2026-07-06 20:59:12.312765427 +0000 UTC m=+601.147652793"
Jul 06 20:59:12.328332 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:12.328295    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8df697c9f-shs9j" podStartSLOduration=2.298354957 podStartE2EDuration="6.328278393s" totalImagesPullingTime="4.029923436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:12.32816745 +0000 UTC m=+601.163054817" watchObservedRunningTime="2026-07-06 20:59:12.328278393 +0000 UTC m=+601.163165761"
Jul 06 20:59:23.300666 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:23.300627    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-69574b596d-nqg8v"
Jul 06 20:59:35.493023 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:35.492933    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-shs9j"]
Jul 06 20:59:35.493586 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:35.493136    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-8df697c9f-shs9j" podUID="f613c345-9ef3-4d95-97f8-68257a9805c6" containerName="authorino" containerID="cri-o://e94ce9848345f4a2d5adeae6908a2094f64ea0e43da981c7eb691eb4156b29eb" gracePeriod=30
Jul 06 20:59:35.734097 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:35.734074    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-shs9j"
Jul 06 20:59:35.810404 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:35.810367    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kngpg\" (UniqueName: \"kubernetes.io/projected/f613c345-9ef3-4d95-97f8-68257a9805c6-kube-api-access-kngpg\") pod \"f613c345-9ef3-4d95-97f8-68257a9805c6\" (UID: \"f613c345-9ef3-4d95-97f8-68257a9805c6\") "
Jul 06 20:59:35.812547 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:35.812516    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f613c345-9ef3-4d95-97f8-68257a9805c6-kube-api-access-kngpg" (OuterVolumeSpecName: "kube-api-access-kngpg") pod "f613c345-9ef3-4d95-97f8-68257a9805c6" (UID: "f613c345-9ef3-4d95-97f8-68257a9805c6"). InnerVolumeSpecName "kube-api-access-kngpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:59:35.911745 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:35.911700    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kngpg\" (UniqueName: \"kubernetes.io/projected/f613c345-9ef3-4d95-97f8-68257a9805c6-kube-api-access-kngpg\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 20:59:36.402819 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:36.402789    2559 generic.go:356] "Generic (PLEG): container finished" podID="f613c345-9ef3-4d95-97f8-68257a9805c6" containerID="e94ce9848345f4a2d5adeae6908a2094f64ea0e43da981c7eb691eb4156b29eb" exitCode=0
Jul 06 20:59:36.402998 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:36.402834    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-shs9j"
Jul 06 20:59:36.402998 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:36.402854    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-shs9j" event={"ID":"f613c345-9ef3-4d95-97f8-68257a9805c6","Type":"ContainerDied","Data":"e94ce9848345f4a2d5adeae6908a2094f64ea0e43da981c7eb691eb4156b29eb"}
Jul 06 20:59:36.402998 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:36.402884    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-shs9j" event={"ID":"f613c345-9ef3-4d95-97f8-68257a9805c6","Type":"ContainerDied","Data":"7fa9339995cef49cb42b66ac26e0128aab63bfc51a55b50742adea0a9e6e7392"}
Jul 06 20:59:36.402998 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:36.402901    2559 scope.go:117] "RemoveContainer" containerID="e94ce9848345f4a2d5adeae6908a2094f64ea0e43da981c7eb691eb4156b29eb"
Jul 06 20:59:36.411881 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:36.411866    2559 scope.go:117] "RemoveContainer" containerID="e94ce9848345f4a2d5adeae6908a2094f64ea0e43da981c7eb691eb4156b29eb"
Jul 06 20:59:36.412132 ip-10-0-134-202 kubenswrapper[2559]: E0706 20:59:36.412110    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94ce9848345f4a2d5adeae6908a2094f64ea0e43da981c7eb691eb4156b29eb\": container with ID starting with e94ce9848345f4a2d5adeae6908a2094f64ea0e43da981c7eb691eb4156b29eb not found: ID does not exist" containerID="e94ce9848345f4a2d5adeae6908a2094f64ea0e43da981c7eb691eb4156b29eb"
Jul 06 20:59:36.412183 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:36.412140    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94ce9848345f4a2d5adeae6908a2094f64ea0e43da981c7eb691eb4156b29eb"} err="failed to get container status \"e94ce9848345f4a2d5adeae6908a2094f64ea0e43da981c7eb691eb4156b29eb\": rpc error: code = NotFound desc = could not find container \"e94ce9848345f4a2d5adeae6908a2094f64ea0e43da981c7eb691eb4156b29eb\": container with ID starting with e94ce9848345f4a2d5adeae6908a2094f64ea0e43da981c7eb691eb4156b29eb not found: ID does not exist"
Jul 06 20:59:36.430291 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:36.430261    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-shs9j"]
Jul 06 20:59:36.432700 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:36.432682    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8df697c9f-shs9j"]
Jul 06 20:59:37.660591 ip-10-0-134-202 kubenswrapper[2559]: I0706 20:59:37.660554    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f613c345-9ef3-4d95-97f8-68257a9805c6" path="/var/lib/kubelet/pods/f613c345-9ef3-4d95-97f8-68257a9805c6/volumes"
Jul 06 21:02:23.967903 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:23.967863    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"]
Jul 06 21:02:23.968366 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:23.968226    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f613c345-9ef3-4d95-97f8-68257a9805c6" containerName="authorino"
Jul 06 21:02:23.968366 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:23.968237    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f613c345-9ef3-4d95-97f8-68257a9805c6" containerName="authorino"
Jul 06 21:02:23.968366 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:23.968308    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f613c345-9ef3-4d95-97f8-68257a9805c6" containerName="authorino"
Jul 06 21:02:23.970259 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:23.970245    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:23.973739 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:23.973716    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g6qfk\""
Jul 06 21:02:23.974065 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:23.974046    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 06 21:02:23.974141 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:23.974094    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:02:23.974200 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:23.974182    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:02:23.974256 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:23.974200    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:02:23.986494 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:23.986472    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"]
Jul 06 21:02:24.035552 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.035523    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-home\") pod \"precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:24.035734 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.035560    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-dshm\") pod \"precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:24.035734 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.035585    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:24.035734 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.035714    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:24.035859 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.035766    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-model-cache\") pod \"precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:24.035894 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.035864    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84xgh\" (UniqueName: \"kubernetes.io/projected/e154fbc4-bd7c-4a47-b605-9a1863c695d0-kube-api-access-84xgh\") pod \"precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:24.035933 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.035919    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e154fbc4-bd7c-4a47-b605-9a1863c695d0-tls-certs\") pod \"precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:24.137263 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.137228    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:24.137460 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.137274    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-model-cache\") pod \"precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:24.137460 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.137313    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84xgh\" (UniqueName: \"kubernetes.io/projected/e154fbc4-bd7c-4a47-b605-9a1863c695d0-kube-api-access-84xgh\") pod \"precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:24.137460 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.137353    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e154fbc4-bd7c-4a47-b605-9a1863c695d0-tls-certs\") pod \"precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:24.137460 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.137390    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-home\") pod \"precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:24.137460 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.137415    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-dshm\") pod \"precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:24.137460 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.137451    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:24.137800 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.137761    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:24.137800 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.137781    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-model-cache\") pod \"precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:24.137952 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.137842    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-home\") pod \"precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:24.137952 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.137853    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:24.139559 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.139542    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-dshm\") pod \"precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:24.139901 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.139886    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e154fbc4-bd7c-4a47-b605-9a1863c695d0-tls-certs\") pod \"precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:24.146625 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.146593    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84xgh\" (UniqueName: \"kubernetes.io/projected/e154fbc4-bd7c-4a47-b605-9a1863c695d0-kube-api-access-84xgh\") pod \"precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:24.280322 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.280224    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:24.427389 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.427361    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"]
Jul 06 21:02:24.428685 ip-10-0-134-202 kubenswrapper[2559]: W0706 21:02:24.428656    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode154fbc4_bd7c_4a47_b605_9a1863c695d0.slice/crio-46fef585764818bea530c3f08a799788779d246482afeee91090305d5e5d7a21 WatchSource:0}: Error finding container 46fef585764818bea530c3f08a799788779d246482afeee91090305d5e5d7a21: Status 404 returned error can't find the container with id 46fef585764818bea530c3f08a799788779d246482afeee91090305d5e5d7a21
Jul 06 21:02:24.430702 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:24.430682    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 21:02:25.165030 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:25.164987    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6" event={"ID":"e154fbc4-bd7c-4a47-b605-9a1863c695d0","Type":"ContainerStarted","Data":"46fef585764818bea530c3f08a799788779d246482afeee91090305d5e5d7a21"}
Jul 06 21:02:29.189197 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:29.189155    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6" event={"ID":"e154fbc4-bd7c-4a47-b605-9a1863c695d0","Type":"ContainerStarted","Data":"5dc909fb36fbf316a860c1741c07832c403213728a92f9de240638761bcd93df"}
Jul 06 21:02:30.801409 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:30.801373    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"]
Jul 06 21:02:30.805279 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:30.805240    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"
Jul 06 21:02:30.807624 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:30.807593    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-epp-sa-dockercfg-jp7c6\""
Jul 06 21:02:30.808539 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:30.808513    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 06 21:02:30.821306 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:30.821281    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"]
Jul 06 21:02:30.897995 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:30.897956    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4696029c-bd97-4850-8ec1-9e590ef9f744-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2\" (UID: \"4696029c-bd97-4850-8ec1-9e590ef9f744\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"
Jul 06 21:02:30.898195 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:30.898002    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9zjw\" (UniqueName: \"kubernetes.io/projected/4696029c-bd97-4850-8ec1-9e590ef9f744-kube-api-access-m9zjw\") pod \"auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2\" (UID: \"4696029c-bd97-4850-8ec1-9e590ef9f744\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"
Jul 06 21:02:30.898195 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:30.898023    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4696029c-bd97-4850-8ec1-9e590ef9f744-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2\" (UID: \"4696029c-bd97-4850-8ec1-9e590ef9f744\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"
Jul 06 21:02:30.898195 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:30.898098    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4696029c-bd97-4850-8ec1-9e590ef9f744-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2\" (UID: \"4696029c-bd97-4850-8ec1-9e590ef9f744\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"
Jul 06 21:02:30.998882 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:30.998760    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4696029c-bd97-4850-8ec1-9e590ef9f744-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2\" (UID: \"4696029c-bd97-4850-8ec1-9e590ef9f744\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"
Jul 06 21:02:30.999170 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:30.998965    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9zjw\" (UniqueName: \"kubernetes.io/projected/4696029c-bd97-4850-8ec1-9e590ef9f744-kube-api-access-m9zjw\") pod \"auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2\" (UID: \"4696029c-bd97-4850-8ec1-9e590ef9f744\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"
Jul 06 21:02:30.999170 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:30.998990    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4696029c-bd97-4850-8ec1-9e590ef9f744-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2\" (UID: \"4696029c-bd97-4850-8ec1-9e590ef9f744\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"
Jul 06 21:02:30.999170 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:30.999007    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4696029c-bd97-4850-8ec1-9e590ef9f744-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2\" (UID: \"4696029c-bd97-4850-8ec1-9e590ef9f744\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"
Jul 06 21:02:30.999515 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:30.999479    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4696029c-bd97-4850-8ec1-9e590ef9f744-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2\" (UID: \"4696029c-bd97-4850-8ec1-9e590ef9f744\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"
Jul 06 21:02:30.999582 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:30.999513    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4696029c-bd97-4850-8ec1-9e590ef9f744-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2\" (UID: \"4696029c-bd97-4850-8ec1-9e590ef9f744\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"
Jul 06 21:02:31.001680 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:31.001652    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4696029c-bd97-4850-8ec1-9e590ef9f744-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2\" (UID: \"4696029c-bd97-4850-8ec1-9e590ef9f744\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"
Jul 06 21:02:31.010244 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:31.010204    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9zjw\" (UniqueName: \"kubernetes.io/projected/4696029c-bd97-4850-8ec1-9e590ef9f744-kube-api-access-m9zjw\") pod \"auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2\" (UID: \"4696029c-bd97-4850-8ec1-9e590ef9f744\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"
Jul 06 21:02:31.123587 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:31.123498    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"
Jul 06 21:02:31.278599 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:31.278564    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"]
Jul 06 21:02:31.281204 ip-10-0-134-202 kubenswrapper[2559]: W0706 21:02:31.281180    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4696029c_bd97_4850_8ec1_9e590ef9f744.slice/crio-ea8c39cb3cdb137f71636068fbb01a16387440687c47fd7a32ccf8172177edf4 WatchSource:0}: Error finding container ea8c39cb3cdb137f71636068fbb01a16387440687c47fd7a32ccf8172177edf4: Status 404 returned error can't find the container with id ea8c39cb3cdb137f71636068fbb01a16387440687c47fd7a32ccf8172177edf4
Jul 06 21:02:32.205410 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:32.205373    2559 generic.go:356] "Generic (PLEG): container finished" podID="e154fbc4-bd7c-4a47-b605-9a1863c695d0" containerID="5dc909fb36fbf316a860c1741c07832c403213728a92f9de240638761bcd93df" exitCode=0
Jul 06 21:02:32.205823 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:32.205442    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6" event={"ID":"e154fbc4-bd7c-4a47-b605-9a1863c695d0","Type":"ContainerDied","Data":"5dc909fb36fbf316a860c1741c07832c403213728a92f9de240638761bcd93df"}
Jul 06 21:02:32.207086 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:32.207064    2559 generic.go:356] "Generic (PLEG): container finished" podID="4696029c-bd97-4850-8ec1-9e590ef9f744" containerID="ece6130cac9c079730cb364a17488122a3e4e8350da154ee37f1eaed79e6041b" exitCode=0
Jul 06 21:02:32.207165 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:32.207113    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2" event={"ID":"4696029c-bd97-4850-8ec1-9e590ef9f744","Type":"ContainerDied","Data":"ece6130cac9c079730cb364a17488122a3e4e8350da154ee37f1eaed79e6041b"}
Jul 06 21:02:32.207165 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:32.207153    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2" event={"ID":"4696029c-bd97-4850-8ec1-9e590ef9f744","Type":"ContainerStarted","Data":"ea8c39cb3cdb137f71636068fbb01a16387440687c47fd7a32ccf8172177edf4"}
Jul 06 21:02:35.223737 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:35.223695    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6" event={"ID":"e154fbc4-bd7c-4a47-b605-9a1863c695d0","Type":"ContainerStarted","Data":"641192bb8cc7dd22b741d22a8dead5cc0f048811cf56d28eeb31f2c2a98ec588"}
Jul 06 21:02:35.225393 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:35.225372    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2" event={"ID":"4696029c-bd97-4850-8ec1-9e590ef9f744","Type":"ContainerStarted","Data":"529ef734e300ac0a87f123e45360de84313eb4e7d1d271beeaea2aca743480ae"}
Jul 06 21:02:35.225502 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:35.225419    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"
Jul 06 21:02:35.253551 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:35.253502    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6" podStartSLOduration=2.728135364 podStartE2EDuration="12.253484678s" totalImagesPullingTime="6.339411947s" totalInitContainerRuntime="3.185937367s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:02:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:02:35.250972485 +0000 UTC m=+804.085859850" watchObservedRunningTime="2026-07-06 21:02:35.253484678 +0000 UTC m=+804.088372044"
Jul 06 21:02:35.274356 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:35.274311    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2" podStartSLOduration=2.165184771 podStartE2EDuration="5.274295401s" totalImagesPullingTime="2.468506143s" totalInitContainerRuntime="640.604487ms" isStatefulPod=false podCreationTimestamp="2026-07-06 21:02:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:02:35.273028863 +0000 UTC m=+804.107916250" watchObservedRunningTime="2026-07-06 21:02:35.274295401 +0000 UTC m=+804.109182766"
Jul 06 21:02:35.280758 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:35.280736    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:44.281023 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:44.280995    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:44.293381 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:44.293357    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:02:45.281578 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:02:45.281552    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:03:06.231897 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:06.231869    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"
Jul 06 21:03:36.709014 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:36.708979    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"]
Jul 06 21:03:51.728882 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:51.728835    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6" podUID="e154fbc4-bd7c-4a47-b605-9a1863c695d0" containerName="main" containerID="cri-o://641192bb8cc7dd22b741d22a8dead5cc0f048811cf56d28eeb31f2c2a98ec588" gracePeriod=45
Jul 06 21:03:51.974843 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:51.974819    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:03:52.016770 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.016694    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-dshm\") pod \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") "
Jul 06 21:03:52.016770 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.016734    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84xgh\" (UniqueName: \"kubernetes.io/projected/e154fbc4-bd7c-4a47-b605-9a1863c695d0-kube-api-access-84xgh\") pod \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") "
Jul 06 21:03:52.016981 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.016780    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-model-cache\") pod \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") "
Jul 06 21:03:52.016981 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.016818    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e154fbc4-bd7c-4a47-b605-9a1863c695d0-tls-certs\") pod \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") "
Jul 06 21:03:52.016981 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.016855    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-tmp-dir\") pod \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") "
Jul 06 21:03:52.016981 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.016910    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-home\") pod \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") "
Jul 06 21:03:52.016981 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.016937    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-kserve-provision-location\") pod \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\" (UID: \"e154fbc4-bd7c-4a47-b605-9a1863c695d0\") "
Jul 06 21:03:52.017216 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.017098    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "e154fbc4-bd7c-4a47-b605-9a1863c695d0" (UID: "e154fbc4-bd7c-4a47-b605-9a1863c695d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:03:52.017216 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.017175    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-home" (OuterVolumeSpecName: "home") pod "e154fbc4-bd7c-4a47-b605-9a1863c695d0" (UID: "e154fbc4-bd7c-4a47-b605-9a1863c695d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:03:52.017325 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.017306    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-model-cache\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:03:52.017382 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.017327    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-home\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:03:52.017582 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.017558    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e154fbc4-bd7c-4a47-b605-9a1863c695d0" (UID: "e154fbc4-bd7c-4a47-b605-9a1863c695d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:03:52.018934 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.018911    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-dshm" (OuterVolumeSpecName: "dshm") pod "e154fbc4-bd7c-4a47-b605-9a1863c695d0" (UID: "e154fbc4-bd7c-4a47-b605-9a1863c695d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:03:52.019339 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.019314    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e154fbc4-bd7c-4a47-b605-9a1863c695d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e154fbc4-bd7c-4a47-b605-9a1863c695d0" (UID: "e154fbc4-bd7c-4a47-b605-9a1863c695d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:03:52.019339 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.019324    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e154fbc4-bd7c-4a47-b605-9a1863c695d0-kube-api-access-84xgh" (OuterVolumeSpecName: "kube-api-access-84xgh") pod "e154fbc4-bd7c-4a47-b605-9a1863c695d0" (UID: "e154fbc4-bd7c-4a47-b605-9a1863c695d0"). InnerVolumeSpecName "kube-api-access-84xgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:03:52.070843 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.070809    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e154fbc4-bd7c-4a47-b605-9a1863c695d0" (UID: "e154fbc4-bd7c-4a47-b605-9a1863c695d0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:03:52.118698 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.118672    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e154fbc4-bd7c-4a47-b605-9a1863c695d0-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:03:52.118698 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.118696    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-tmp-dir\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:03:52.118833 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.118705    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-kserve-provision-location\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:03:52.118833 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.118714    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e154fbc4-bd7c-4a47-b605-9a1863c695d0-dshm\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:03:52.118833 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.118723    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84xgh\" (UniqueName: \"kubernetes.io/projected/e154fbc4-bd7c-4a47-b605-9a1863c695d0-kube-api-access-84xgh\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:03:52.587180 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.587150    2559 generic.go:356] "Generic (PLEG): container finished" podID="e154fbc4-bd7c-4a47-b605-9a1863c695d0" containerID="641192bb8cc7dd22b741d22a8dead5cc0f048811cf56d28eeb31f2c2a98ec588" exitCode=0
Jul 06 21:03:52.587359 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.587191    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6" event={"ID":"e154fbc4-bd7c-4a47-b605-9a1863c695d0","Type":"ContainerDied","Data":"641192bb8cc7dd22b741d22a8dead5cc0f048811cf56d28eeb31f2c2a98ec588"}
Jul 06 21:03:52.587359 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.587202    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"
Jul 06 21:03:52.587359 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.587232    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6" event={"ID":"e154fbc4-bd7c-4a47-b605-9a1863c695d0","Type":"ContainerDied","Data":"46fef585764818bea530c3f08a799788779d246482afeee91090305d5e5d7a21"}
Jul 06 21:03:52.587359 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.587254    2559 scope.go:117] "RemoveContainer" containerID="641192bb8cc7dd22b741d22a8dead5cc0f048811cf56d28eeb31f2c2a98ec588"
Jul 06 21:03:52.596772 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.596755    2559 scope.go:117] "RemoveContainer" containerID="5dc909fb36fbf316a860c1741c07832c403213728a92f9de240638761bcd93df"
Jul 06 21:03:52.604466 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.604449    2559 scope.go:117] "RemoveContainer" containerID="641192bb8cc7dd22b741d22a8dead5cc0f048811cf56d28eeb31f2c2a98ec588"
Jul 06 21:03:52.604703 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:03:52.604688    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"641192bb8cc7dd22b741d22a8dead5cc0f048811cf56d28eeb31f2c2a98ec588\": container with ID starting with 641192bb8cc7dd22b741d22a8dead5cc0f048811cf56d28eeb31f2c2a98ec588 not found: ID does not exist" containerID="641192bb8cc7dd22b741d22a8dead5cc0f048811cf56d28eeb31f2c2a98ec588"
Jul 06 21:03:52.604749 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.604712    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"641192bb8cc7dd22b741d22a8dead5cc0f048811cf56d28eeb31f2c2a98ec588"} err="failed to get container status \"641192bb8cc7dd22b741d22a8dead5cc0f048811cf56d28eeb31f2c2a98ec588\": rpc error: code = NotFound desc = could not find container \"641192bb8cc7dd22b741d22a8dead5cc0f048811cf56d28eeb31f2c2a98ec588\": container with ID starting with 641192bb8cc7dd22b741d22a8dead5cc0f048811cf56d28eeb31f2c2a98ec588 not found: ID does not exist"
Jul 06 21:03:52.604749 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.604728    2559 scope.go:117] "RemoveContainer" containerID="5dc909fb36fbf316a860c1741c07832c403213728a92f9de240638761bcd93df"
Jul 06 21:03:52.604959 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:03:52.604941    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dc909fb36fbf316a860c1741c07832c403213728a92f9de240638761bcd93df\": container with ID starting with 5dc909fb36fbf316a860c1741c07832c403213728a92f9de240638761bcd93df not found: ID does not exist" containerID="5dc909fb36fbf316a860c1741c07832c403213728a92f9de240638761bcd93df"
Jul 06 21:03:52.605001 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.604963    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dc909fb36fbf316a860c1741c07832c403213728a92f9de240638761bcd93df"} err="failed to get container status \"5dc909fb36fbf316a860c1741c07832c403213728a92f9de240638761bcd93df\": rpc error: code = NotFound desc = could not find container \"5dc909fb36fbf316a860c1741c07832c403213728a92f9de240638761bcd93df\": container with ID starting with 5dc909fb36fbf316a860c1741c07832c403213728a92f9de240638761bcd93df not found: ID does not exist"
Jul 06 21:03:52.611293 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.611272    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"]
Jul 06 21:03:52.612950 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:52.612931    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6"]
Jul 06 21:03:53.658081 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:03:53.658048    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e154fbc4-bd7c-4a47-b605-9a1863c695d0" path="/var/lib/kubelet/pods/e154fbc4-bd7c-4a47-b605-9a1863c695d0/volumes"
Jul 06 21:05:20.177044 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.177014    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"]
Jul 06 21:05:20.185642 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:05:20.185588    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:05:20.185642 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:05:20.185642 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 21:05:20.185642 ip-10-0-134-202 kubenswrapper[2559]:  > containerID="529ef734e300ac0a87f123e45360de84313eb4e7d1d271beeaea2aca743480ae" cmd=["/bin/sleep","15"]
Jul 06 21:05:20.185868 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.185664    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2" message="" err=<
Jul 06 21:05:20.185868 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:05:20.185868 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 21:05:20.185868 ip-10-0-134-202 kubenswrapper[2559]:  >
Jul 06 21:05:20.185868 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:05:20.185679    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:05:20.185868 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:05:20.185868 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 21:05:20.185868 ip-10-0-134-202 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2" podUID="4696029c-bd97-4850-8ec1-9e590ef9f744" containerName="main" containerID="cri-o://529ef734e300ac0a87f123e45360de84313eb4e7d1d271beeaea2aca743480ae"
Jul 06 21:05:20.185868 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.185723    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2" podUID="4696029c-bd97-4850-8ec1-9e590ef9f744" containerName="main" containerID="cri-o://529ef734e300ac0a87f123e45360de84313eb4e7d1d271beeaea2aca743480ae" gracePeriod=60
Jul 06 21:05:20.430338 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.430288    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"
Jul 06 21:05:20.499704 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.499668    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4696029c-bd97-4850-8ec1-9e590ef9f744-tls-certs\") pod \"4696029c-bd97-4850-8ec1-9e590ef9f744\" (UID: \"4696029c-bd97-4850-8ec1-9e590ef9f744\") "
Jul 06 21:05:20.499891 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.499754    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9zjw\" (UniqueName: \"kubernetes.io/projected/4696029c-bd97-4850-8ec1-9e590ef9f744-kube-api-access-m9zjw\") pod \"4696029c-bd97-4850-8ec1-9e590ef9f744\" (UID: \"4696029c-bd97-4850-8ec1-9e590ef9f744\") "
Jul 06 21:05:20.499891 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.499783    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4696029c-bd97-4850-8ec1-9e590ef9f744-tokenizer-uds\") pod \"4696029c-bd97-4850-8ec1-9e590ef9f744\" (UID: \"4696029c-bd97-4850-8ec1-9e590ef9f744\") "
Jul 06 21:05:20.499891 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.499839    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4696029c-bd97-4850-8ec1-9e590ef9f744-kserve-provision-location\") pod \"4696029c-bd97-4850-8ec1-9e590ef9f744\" (UID: \"4696029c-bd97-4850-8ec1-9e590ef9f744\") "
Jul 06 21:05:20.500147 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.500121    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4696029c-bd97-4850-8ec1-9e590ef9f744-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "4696029c-bd97-4850-8ec1-9e590ef9f744" (UID: "4696029c-bd97-4850-8ec1-9e590ef9f744"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:05:20.500599 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.500577    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4696029c-bd97-4850-8ec1-9e590ef9f744-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4696029c-bd97-4850-8ec1-9e590ef9f744" (UID: "4696029c-bd97-4850-8ec1-9e590ef9f744"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:05:20.501723 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.501698    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4696029c-bd97-4850-8ec1-9e590ef9f744-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4696029c-bd97-4850-8ec1-9e590ef9f744" (UID: "4696029c-bd97-4850-8ec1-9e590ef9f744"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:05:20.501806 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.501792    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4696029c-bd97-4850-8ec1-9e590ef9f744-kube-api-access-m9zjw" (OuterVolumeSpecName: "kube-api-access-m9zjw") pod "4696029c-bd97-4850-8ec1-9e590ef9f744" (UID: "4696029c-bd97-4850-8ec1-9e590ef9f744"). InnerVolumeSpecName "kube-api-access-m9zjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:05:20.600927 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.600894    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4696029c-bd97-4850-8ec1-9e590ef9f744-tokenizer-uds\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:05:20.600927 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.600920    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4696029c-bd97-4850-8ec1-9e590ef9f744-kserve-provision-location\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:05:20.600927 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.600931    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4696029c-bd97-4850-8ec1-9e590ef9f744-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:05:20.601174 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.600942    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9zjw\" (UniqueName: \"kubernetes.io/projected/4696029c-bd97-4850-8ec1-9e590ef9f744-kube-api-access-m9zjw\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:05:20.980920 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.980890    2559 generic.go:356] "Generic (PLEG): container finished" podID="4696029c-bd97-4850-8ec1-9e590ef9f744" containerID="529ef734e300ac0a87f123e45360de84313eb4e7d1d271beeaea2aca743480ae" exitCode=0
Jul 06 21:05:20.981108 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.980947    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"
Jul 06 21:05:20.981108 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.980978    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2" event={"ID":"4696029c-bd97-4850-8ec1-9e590ef9f744","Type":"ContainerDied","Data":"529ef734e300ac0a87f123e45360de84313eb4e7d1d271beeaea2aca743480ae"}
Jul 06 21:05:20.981108 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.981031    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2" event={"ID":"4696029c-bd97-4850-8ec1-9e590ef9f744","Type":"ContainerDied","Data":"ea8c39cb3cdb137f71636068fbb01a16387440687c47fd7a32ccf8172177edf4"}
Jul 06 21:05:20.981108 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.981051    2559 scope.go:117] "RemoveContainer" containerID="529ef734e300ac0a87f123e45360de84313eb4e7d1d271beeaea2aca743480ae"
Jul 06 21:05:20.990389 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.990374    2559 scope.go:117] "RemoveContainer" containerID="ece6130cac9c079730cb364a17488122a3e4e8350da154ee37f1eaed79e6041b"
Jul 06 21:05:20.998435 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.998416    2559 scope.go:117] "RemoveContainer" containerID="529ef734e300ac0a87f123e45360de84313eb4e7d1d271beeaea2aca743480ae"
Jul 06 21:05:20.998691 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:05:20.998673    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"529ef734e300ac0a87f123e45360de84313eb4e7d1d271beeaea2aca743480ae\": container with ID starting with 529ef734e300ac0a87f123e45360de84313eb4e7d1d271beeaea2aca743480ae not found: ID does not exist" containerID="529ef734e300ac0a87f123e45360de84313eb4e7d1d271beeaea2aca743480ae"
Jul 06 21:05:20.998768 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.998702    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"529ef734e300ac0a87f123e45360de84313eb4e7d1d271beeaea2aca743480ae"} err="failed to get container status \"529ef734e300ac0a87f123e45360de84313eb4e7d1d271beeaea2aca743480ae\": rpc error: code = NotFound desc = could not find container \"529ef734e300ac0a87f123e45360de84313eb4e7d1d271beeaea2aca743480ae\": container with ID starting with 529ef734e300ac0a87f123e45360de84313eb4e7d1d271beeaea2aca743480ae not found: ID does not exist"
Jul 06 21:05:20.998768 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.998726    2559 scope.go:117] "RemoveContainer" containerID="ece6130cac9c079730cb364a17488122a3e4e8350da154ee37f1eaed79e6041b"
Jul 06 21:05:20.998983 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:05:20.998967    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ece6130cac9c079730cb364a17488122a3e4e8350da154ee37f1eaed79e6041b\": container with ID starting with ece6130cac9c079730cb364a17488122a3e4e8350da154ee37f1eaed79e6041b not found: ID does not exist" containerID="ece6130cac9c079730cb364a17488122a3e4e8350da154ee37f1eaed79e6041b"
Jul 06 21:05:20.999019 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:20.998989    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ece6130cac9c079730cb364a17488122a3e4e8350da154ee37f1eaed79e6041b"} err="failed to get container status \"ece6130cac9c079730cb364a17488122a3e4e8350da154ee37f1eaed79e6041b\": rpc error: code = NotFound desc = could not find container \"ece6130cac9c079730cb364a17488122a3e4e8350da154ee37f1eaed79e6041b\": container with ID starting with ece6130cac9c079730cb364a17488122a3e4e8350da154ee37f1eaed79e6041b not found: ID does not exist"
Jul 06 21:05:21.003332 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:21.003312    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"]
Jul 06 21:05:21.008056 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:21.008035    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2"]
Jul 06 21:05:21.657334 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:21.657299    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4696029c-bd97-4850-8ec1-9e590ef9f744" path="/var/lib/kubelet/pods/4696029c-bd97-4850-8ec1-9e590ef9f744/volumes"
Jul 06 21:05:42.282802 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.282723    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"]
Jul 06 21:05:42.283198 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.283086    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4696029c-bd97-4850-8ec1-9e590ef9f744" containerName="storage-initializer"
Jul 06 21:05:42.283198 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.283098    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4696029c-bd97-4850-8ec1-9e590ef9f744" containerName="storage-initializer"
Jul 06 21:05:42.283198 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.283119    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4696029c-bd97-4850-8ec1-9e590ef9f744" containerName="main"
Jul 06 21:05:42.283198 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.283125    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4696029c-bd97-4850-8ec1-9e590ef9f744" containerName="main"
Jul 06 21:05:42.283198 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.283133    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e154fbc4-bd7c-4a47-b605-9a1863c695d0" containerName="main"
Jul 06 21:05:42.283198 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.283138    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e154fbc4-bd7c-4a47-b605-9a1863c695d0" containerName="main"
Jul 06 21:05:42.283198 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.283148    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e154fbc4-bd7c-4a47-b605-9a1863c695d0" containerName="storage-initializer"
Jul 06 21:05:42.283198 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.283153    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e154fbc4-bd7c-4a47-b605-9a1863c695d0" containerName="storage-initializer"
Jul 06 21:05:42.283430 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.283205    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4696029c-bd97-4850-8ec1-9e590ef9f744" containerName="main"
Jul 06 21:05:42.283430 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.283218    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e154fbc4-bd7c-4a47-b605-9a1863c695d0" containerName="main"
Jul 06 21:05:42.285131 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.285115    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"
Jul 06 21:05:42.287543 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.287519    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:05:42.287812 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.287797    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g6qfk\""
Jul 06 21:05:42.287871 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.287815    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:05:42.288547 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.288525    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-epp-sa-dockercfg-zb4jd\""
Jul 06 21:05:42.288654 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.288553    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:05:42.288654 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.288595    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 06 21:05:42.304791 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.304768    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"]
Jul 06 21:05:42.384549 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.384514    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40813f75-5b7b-497f-9477-0daa7dadfafc-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx\" (UID: \"40813f75-5b7b-497f-9477-0daa7dadfafc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"
Jul 06 21:05:42.384549 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.384551    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/40813f75-5b7b-497f-9477-0daa7dadfafc-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx\" (UID: \"40813f75-5b7b-497f-9477-0daa7dadfafc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"
Jul 06 21:05:42.384813 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.384573    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h5nj\" (UniqueName: \"kubernetes.io/projected/40813f75-5b7b-497f-9477-0daa7dadfafc-kube-api-access-8h5nj\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx\" (UID: \"40813f75-5b7b-497f-9477-0daa7dadfafc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"
Jul 06 21:05:42.384813 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.384676    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40813f75-5b7b-497f-9477-0daa7dadfafc-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx\" (UID: \"40813f75-5b7b-497f-9477-0daa7dadfafc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"
Jul 06 21:05:42.485735 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.485695    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40813f75-5b7b-497f-9477-0daa7dadfafc-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx\" (UID: \"40813f75-5b7b-497f-9477-0daa7dadfafc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"
Jul 06 21:05:42.485735 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.485732    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/40813f75-5b7b-497f-9477-0daa7dadfafc-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx\" (UID: \"40813f75-5b7b-497f-9477-0daa7dadfafc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"
Jul 06 21:05:42.485984 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.485755    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h5nj\" (UniqueName: \"kubernetes.io/projected/40813f75-5b7b-497f-9477-0daa7dadfafc-kube-api-access-8h5nj\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx\" (UID: \"40813f75-5b7b-497f-9477-0daa7dadfafc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"
Jul 06 21:05:42.485984 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.485777    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40813f75-5b7b-497f-9477-0daa7dadfafc-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx\" (UID: \"40813f75-5b7b-497f-9477-0daa7dadfafc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"
Jul 06 21:05:42.486091 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.486062    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/40813f75-5b7b-497f-9477-0daa7dadfafc-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx\" (UID: \"40813f75-5b7b-497f-9477-0daa7dadfafc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"
Jul 06 21:05:42.486139 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.486092    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40813f75-5b7b-497f-9477-0daa7dadfafc-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx\" (UID: \"40813f75-5b7b-497f-9477-0daa7dadfafc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"
Jul 06 21:05:42.488208 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.488187    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40813f75-5b7b-497f-9477-0daa7dadfafc-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx\" (UID: \"40813f75-5b7b-497f-9477-0daa7dadfafc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"
Jul 06 21:05:42.495938 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.495906    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h5nj\" (UniqueName: \"kubernetes.io/projected/40813f75-5b7b-497f-9477-0daa7dadfafc-kube-api-access-8h5nj\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx\" (UID: \"40813f75-5b7b-497f-9477-0daa7dadfafc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"
Jul 06 21:05:42.593950 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.593919    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"
Jul 06 21:05:42.730552 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:42.730520    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"]
Jul 06 21:05:42.733507 ip-10-0-134-202 kubenswrapper[2559]: W0706 21:05:42.733475    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40813f75_5b7b_497f_9477_0daa7dadfafc.slice/crio-34c9150c253507d5896569c1e063cd6d686c7b7917e1c817a8accbdf373a2f3a WatchSource:0}: Error finding container 34c9150c253507d5896569c1e063cd6d686c7b7917e1c817a8accbdf373a2f3a: Status 404 returned error can't find the container with id 34c9150c253507d5896569c1e063cd6d686c7b7917e1c817a8accbdf373a2f3a
Jul 06 21:05:43.090318 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:43.090264    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx" event={"ID":"40813f75-5b7b-497f-9477-0daa7dadfafc","Type":"ContainerStarted","Data":"a1b60d9af4ce5dc72396be9c20f2d1d59eb9d4bf422c55bc993dcbe7099fd846"}
Jul 06 21:05:43.090533 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:43.090369    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx" event={"ID":"40813f75-5b7b-497f-9477-0daa7dadfafc","Type":"ContainerStarted","Data":"34c9150c253507d5896569c1e063cd6d686c7b7917e1c817a8accbdf373a2f3a"}
Jul 06 21:05:44.095765 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:44.095731    2559 generic.go:356] "Generic (PLEG): container finished" podID="40813f75-5b7b-497f-9477-0daa7dadfafc" containerID="a1b60d9af4ce5dc72396be9c20f2d1d59eb9d4bf422c55bc993dcbe7099fd846" exitCode=0
Jul 06 21:05:44.096150 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:44.095818    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx" event={"ID":"40813f75-5b7b-497f-9477-0daa7dadfafc","Type":"ContainerDied","Data":"a1b60d9af4ce5dc72396be9c20f2d1d59eb9d4bf422c55bc993dcbe7099fd846"}
Jul 06 21:05:45.101673 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:45.101640    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx" event={"ID":"40813f75-5b7b-497f-9477-0daa7dadfafc","Type":"ContainerStarted","Data":"554aa49a633863e9e716d996a5f8ccfc588ced9a92e097ba949912ef48609f4a"}
Jul 06 21:05:45.102054 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:45.101845    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"
Jul 06 21:05:45.125650 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:05:45.125579    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx" podStartSLOduration=2.443194123 podStartE2EDuration="3.12556148s" totalImagesPullingTime="0s" totalInitContainerRuntime="682.367357ms" isStatefulPod=false podCreationTimestamp="2026-07-06 21:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:05:45.124597395 +0000 UTC m=+993.959484760" watchObservedRunningTime="2026-07-06 21:05:45.12556148 +0000 UTC m=+993.960448843"
Jul 06 21:06:10.436784 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.436755    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"]
Jul 06 21:06:10.438966 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.438947    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:10.441387 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.441371    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 06 21:06:10.451093 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.451074    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"]
Jul 06 21:06:10.537432 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.537398    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:10.537432 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.537434    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:10.537689 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.537466    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-954ms\" (UniqueName: \"kubernetes.io/projected/caa807b9-8f0d-4996-905f-99ce948ac7ce-kube-api-access-954ms\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:10.537689 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.537504    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:10.537689 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.537528    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caa807b9-8f0d-4996-905f-99ce948ac7ce-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:10.537689 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.537588    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:10.638761 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.638724    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:10.638927 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.638772    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caa807b9-8f0d-4996-905f-99ce948ac7ce-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:10.638927 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.638884    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:10.639038 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.638976    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:10.639038 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.639010    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:10.639143 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.639054    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-954ms\" (UniqueName: \"kubernetes.io/projected/caa807b9-8f0d-4996-905f-99ce948ac7ce-kube-api-access-954ms\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:10.639143 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.639123    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:10.639345 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.639322    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:10.639425 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.639356    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:10.641110 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.641092    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:10.641352 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.641336    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caa807b9-8f0d-4996-905f-99ce948ac7ce-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:10.649587 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.649566    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-954ms\" (UniqueName: \"kubernetes.io/projected/caa807b9-8f0d-4996-905f-99ce948ac7ce-kube-api-access-954ms\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:10.749105 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.749023    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:10.885691 ip-10-0-134-202 kubenswrapper[2559]: W0706 21:06:10.885662    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa807b9_8f0d_4996_905f_99ce948ac7ce.slice/crio-b07151371735528ffae3266daad898de5f728c2009db95258df795712b1a7286 WatchSource:0}: Error finding container b07151371735528ffae3266daad898de5f728c2009db95258df795712b1a7286: Status 404 returned error can't find the container with id b07151371735528ffae3266daad898de5f728c2009db95258df795712b1a7286
Jul 06 21:06:10.886099 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:10.886072    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"]
Jul 06 21:06:11.216538 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:11.216502    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6" event={"ID":"caa807b9-8f0d-4996-905f-99ce948ac7ce","Type":"ContainerStarted","Data":"72e06444432adf5b3fe4e02964bb9ba7b0a8e85913386a784d88902ec1b598b2"}
Jul 06 21:06:11.216728 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:11.216560    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6" event={"ID":"caa807b9-8f0d-4996-905f-99ce948ac7ce","Type":"ContainerStarted","Data":"b07151371735528ffae3266daad898de5f728c2009db95258df795712b1a7286"}
Jul 06 21:06:11.280570 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:11.280518    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6" podStartSLOduration=1.280503433 podStartE2EDuration="1.280503433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:06:11.274799997 +0000 UTC m=+1020.109687361" watchObservedRunningTime="2026-07-06 21:06:11.280503433 +0000 UTC m=+1020.115390799"
Jul 06 21:06:11.749267 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:11.749236    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:16.108454 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:16.108426    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"
Jul 06 21:06:20.749951 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:20.749913    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:20.762076 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:20.762041    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:21.273327 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:21.273295    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:06:44.849304 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:44.849262    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"]
Jul 06 21:06:59.873591 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:06:59.873551    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6" podUID="caa807b9-8f0d-4996-905f-99ce948ac7ce" containerName="main" containerID="cri-o://72e06444432adf5b3fe4e02964bb9ba7b0a8e85913386a784d88902ec1b598b2" gracePeriod=45
Jul 06 21:07:00.124921 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.124858    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:07:00.277642 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.277591    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caa807b9-8f0d-4996-905f-99ce948ac7ce-tls-certs\") pod \"caa807b9-8f0d-4996-905f-99ce948ac7ce\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") "
Jul 06 21:07:00.277817 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.277681    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-tmp-dir\") pod \"caa807b9-8f0d-4996-905f-99ce948ac7ce\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") "
Jul 06 21:07:00.277817 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.277717    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-model-cache\") pod \"caa807b9-8f0d-4996-905f-99ce948ac7ce\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") "
Jul 06 21:07:00.277817 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.277763    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-home\") pod \"caa807b9-8f0d-4996-905f-99ce948ac7ce\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") "
Jul 06 21:07:00.277817 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.277785    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-dshm\") pod \"caa807b9-8f0d-4996-905f-99ce948ac7ce\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") "
Jul 06 21:07:00.278000 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.277827    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-954ms\" (UniqueName: \"kubernetes.io/projected/caa807b9-8f0d-4996-905f-99ce948ac7ce-kube-api-access-954ms\") pod \"caa807b9-8f0d-4996-905f-99ce948ac7ce\" (UID: \"caa807b9-8f0d-4996-905f-99ce948ac7ce\") "
Jul 06 21:07:00.278000 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.277929    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "caa807b9-8f0d-4996-905f-99ce948ac7ce" (UID: "caa807b9-8f0d-4996-905f-99ce948ac7ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:07:00.278104 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.278050    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-model-cache" (OuterVolumeSpecName: "model-cache") pod "caa807b9-8f0d-4996-905f-99ce948ac7ce" (UID: "caa807b9-8f0d-4996-905f-99ce948ac7ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:07:00.278104 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.278064    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-home" (OuterVolumeSpecName: "home") pod "caa807b9-8f0d-4996-905f-99ce948ac7ce" (UID: "caa807b9-8f0d-4996-905f-99ce948ac7ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:07:00.278219 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.278165    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-tmp-dir\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:07:00.278219 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.278185    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-model-cache\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:07:00.278219 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.278198    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-home\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:07:00.279718 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.279694    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caa807b9-8f0d-4996-905f-99ce948ac7ce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "caa807b9-8f0d-4996-905f-99ce948ac7ce" (UID: "caa807b9-8f0d-4996-905f-99ce948ac7ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:07:00.279945 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.279919    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-dshm" (OuterVolumeSpecName: "dshm") pod "caa807b9-8f0d-4996-905f-99ce948ac7ce" (UID: "caa807b9-8f0d-4996-905f-99ce948ac7ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:07:00.280112 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.280095    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa807b9-8f0d-4996-905f-99ce948ac7ce-kube-api-access-954ms" (OuterVolumeSpecName: "kube-api-access-954ms") pod "caa807b9-8f0d-4996-905f-99ce948ac7ce" (UID: "caa807b9-8f0d-4996-905f-99ce948ac7ce"). InnerVolumeSpecName "kube-api-access-954ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:07:00.379536 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.379451    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caa807b9-8f0d-4996-905f-99ce948ac7ce-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:07:00.379536 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.379481    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa807b9-8f0d-4996-905f-99ce948ac7ce-dshm\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:07:00.379536 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.379493    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-954ms\" (UniqueName: \"kubernetes.io/projected/caa807b9-8f0d-4996-905f-99ce948ac7ce-kube-api-access-954ms\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:07:00.441782 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.441746    2559 generic.go:356] "Generic (PLEG): container finished" podID="caa807b9-8f0d-4996-905f-99ce948ac7ce" containerID="72e06444432adf5b3fe4e02964bb9ba7b0a8e85913386a784d88902ec1b598b2" exitCode=0
Jul 06 21:07:00.441922 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.441787    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"
Jul 06 21:07:00.441922 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.441807    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6" event={"ID":"caa807b9-8f0d-4996-905f-99ce948ac7ce","Type":"ContainerDied","Data":"72e06444432adf5b3fe4e02964bb9ba7b0a8e85913386a784d88902ec1b598b2"}
Jul 06 21:07:00.441922 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.441854    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6" event={"ID":"caa807b9-8f0d-4996-905f-99ce948ac7ce","Type":"ContainerDied","Data":"b07151371735528ffae3266daad898de5f728c2009db95258df795712b1a7286"}
Jul 06 21:07:00.441922 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.441878    2559 scope.go:117] "RemoveContainer" containerID="72e06444432adf5b3fe4e02964bb9ba7b0a8e85913386a784d88902ec1b598b2"
Jul 06 21:07:00.451018 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.451000    2559 scope.go:117] "RemoveContainer" containerID="72e06444432adf5b3fe4e02964bb9ba7b0a8e85913386a784d88902ec1b598b2"
Jul 06 21:07:00.451277 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:07:00.451260    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72e06444432adf5b3fe4e02964bb9ba7b0a8e85913386a784d88902ec1b598b2\": container with ID starting with 72e06444432adf5b3fe4e02964bb9ba7b0a8e85913386a784d88902ec1b598b2 not found: ID does not exist" containerID="72e06444432adf5b3fe4e02964bb9ba7b0a8e85913386a784d88902ec1b598b2"
Jul 06 21:07:00.451323 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.451286    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72e06444432adf5b3fe4e02964bb9ba7b0a8e85913386a784d88902ec1b598b2"} err="failed to get container status \"72e06444432adf5b3fe4e02964bb9ba7b0a8e85913386a784d88902ec1b598b2\": rpc error: code = NotFound desc = could not find container \"72e06444432adf5b3fe4e02964bb9ba7b0a8e85913386a784d88902ec1b598b2\": container with ID starting with 72e06444432adf5b3fe4e02964bb9ba7b0a8e85913386a784d88902ec1b598b2 not found: ID does not exist"
Jul 06 21:07:00.469099 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.469074    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"]
Jul 06 21:07:00.476527 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:00.476506    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6"]
Jul 06 21:07:01.657497 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:01.657456    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caa807b9-8f0d-4996-905f-99ce948ac7ce" path="/var/lib/kubelet/pods/caa807b9-8f0d-4996-905f-99ce948ac7ce/volumes"
Jul 06 21:07:43.803419 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:43.803376    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"]
Jul 06 21:07:43.812398 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:07:43.812364    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:07:43.812398 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:07:43.812398 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 21:07:43.812398 ip-10-0-134-202 kubenswrapper[2559]:  > containerID="554aa49a633863e9e716d996a5f8ccfc588ced9a92e097ba949912ef48609f4a" cmd=["/bin/sleep","15"]
Jul 06 21:07:43.812724 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:43.812410    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx" message="" err=<
Jul 06 21:07:43.812724 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:07:43.812724 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 21:07:43.812724 ip-10-0-134-202 kubenswrapper[2559]:  >
Jul 06 21:07:43.812724 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:07:43.812423    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:07:43.812724 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:07:43.812724 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 21:07:43.812724 ip-10-0-134-202 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx" podUID="40813f75-5b7b-497f-9477-0daa7dadfafc" containerName="main" containerID="cri-o://554aa49a633863e9e716d996a5f8ccfc588ced9a92e097ba949912ef48609f4a"
Jul 06 21:07:43.812724 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:43.812460    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx" podUID="40813f75-5b7b-497f-9477-0daa7dadfafc" containerName="main" containerID="cri-o://554aa49a633863e9e716d996a5f8ccfc588ced9a92e097ba949912ef48609f4a" gracePeriod=60
Jul 06 21:07:44.069434 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.069410    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"
Jul 06 21:07:44.157185 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.157153    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8h5nj\" (UniqueName: \"kubernetes.io/projected/40813f75-5b7b-497f-9477-0daa7dadfafc-kube-api-access-8h5nj\") pod \"40813f75-5b7b-497f-9477-0daa7dadfafc\" (UID: \"40813f75-5b7b-497f-9477-0daa7dadfafc\") "
Jul 06 21:07:44.157336 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.157225    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40813f75-5b7b-497f-9477-0daa7dadfafc-tls-certs\") pod \"40813f75-5b7b-497f-9477-0daa7dadfafc\" (UID: \"40813f75-5b7b-497f-9477-0daa7dadfafc\") "
Jul 06 21:07:44.157336 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.157280    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40813f75-5b7b-497f-9477-0daa7dadfafc-kserve-provision-location\") pod \"40813f75-5b7b-497f-9477-0daa7dadfafc\" (UID: \"40813f75-5b7b-497f-9477-0daa7dadfafc\") "
Jul 06 21:07:44.157336 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.157311    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/40813f75-5b7b-497f-9477-0daa7dadfafc-tokenizer-uds\") pod \"40813f75-5b7b-497f-9477-0daa7dadfafc\" (UID: \"40813f75-5b7b-497f-9477-0daa7dadfafc\") "
Jul 06 21:07:44.157670 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.157639    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40813f75-5b7b-497f-9477-0daa7dadfafc-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "40813f75-5b7b-497f-9477-0daa7dadfafc" (UID: "40813f75-5b7b-497f-9477-0daa7dadfafc"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:07:44.158016 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.157997    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40813f75-5b7b-497f-9477-0daa7dadfafc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "40813f75-5b7b-497f-9477-0daa7dadfafc" (UID: "40813f75-5b7b-497f-9477-0daa7dadfafc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:07:44.159373 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.159352    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40813f75-5b7b-497f-9477-0daa7dadfafc-kube-api-access-8h5nj" (OuterVolumeSpecName: "kube-api-access-8h5nj") pod "40813f75-5b7b-497f-9477-0daa7dadfafc" (UID: "40813f75-5b7b-497f-9477-0daa7dadfafc"). InnerVolumeSpecName "kube-api-access-8h5nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:07:44.159459 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.159391    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40813f75-5b7b-497f-9477-0daa7dadfafc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40813f75-5b7b-497f-9477-0daa7dadfafc" (UID: "40813f75-5b7b-497f-9477-0daa7dadfafc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:07:44.258165 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.258134    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40813f75-5b7b-497f-9477-0daa7dadfafc-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:07:44.258165 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.258161    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40813f75-5b7b-497f-9477-0daa7dadfafc-kserve-provision-location\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:07:44.258165 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.258170    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/40813f75-5b7b-497f-9477-0daa7dadfafc-tokenizer-uds\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:07:44.258395 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.258179    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h5nj\" (UniqueName: \"kubernetes.io/projected/40813f75-5b7b-497f-9477-0daa7dadfafc-kube-api-access-8h5nj\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:07:44.645028 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.644989    2559 generic.go:356] "Generic (PLEG): container finished" podID="40813f75-5b7b-497f-9477-0daa7dadfafc" containerID="554aa49a633863e9e716d996a5f8ccfc588ced9a92e097ba949912ef48609f4a" exitCode=0
Jul 06 21:07:44.645217 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.645071    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"
Jul 06 21:07:44.645217 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.645073    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx" event={"ID":"40813f75-5b7b-497f-9477-0daa7dadfafc","Type":"ContainerDied","Data":"554aa49a633863e9e716d996a5f8ccfc588ced9a92e097ba949912ef48609f4a"}
Jul 06 21:07:44.645217 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.645142    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx" event={"ID":"40813f75-5b7b-497f-9477-0daa7dadfafc","Type":"ContainerDied","Data":"34c9150c253507d5896569c1e063cd6d686c7b7917e1c817a8accbdf373a2f3a"}
Jul 06 21:07:44.645217 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.645170    2559 scope.go:117] "RemoveContainer" containerID="554aa49a633863e9e716d996a5f8ccfc588ced9a92e097ba949912ef48609f4a"
Jul 06 21:07:44.660992 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.660957    2559 scope.go:117] "RemoveContainer" containerID="a1b60d9af4ce5dc72396be9c20f2d1d59eb9d4bf422c55bc993dcbe7099fd846"
Jul 06 21:07:44.668794 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.668780    2559 scope.go:117] "RemoveContainer" containerID="554aa49a633863e9e716d996a5f8ccfc588ced9a92e097ba949912ef48609f4a"
Jul 06 21:07:44.669003 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:07:44.668986    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"554aa49a633863e9e716d996a5f8ccfc588ced9a92e097ba949912ef48609f4a\": container with ID starting with 554aa49a633863e9e716d996a5f8ccfc588ced9a92e097ba949912ef48609f4a not found: ID does not exist" containerID="554aa49a633863e9e716d996a5f8ccfc588ced9a92e097ba949912ef48609f4a"
Jul 06 21:07:44.669053 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.669010    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"554aa49a633863e9e716d996a5f8ccfc588ced9a92e097ba949912ef48609f4a"} err="failed to get container status \"554aa49a633863e9e716d996a5f8ccfc588ced9a92e097ba949912ef48609f4a\": rpc error: code = NotFound desc = could not find container \"554aa49a633863e9e716d996a5f8ccfc588ced9a92e097ba949912ef48609f4a\": container with ID starting with 554aa49a633863e9e716d996a5f8ccfc588ced9a92e097ba949912ef48609f4a not found: ID does not exist"
Jul 06 21:07:44.669053 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.669024    2559 scope.go:117] "RemoveContainer" containerID="a1b60d9af4ce5dc72396be9c20f2d1d59eb9d4bf422c55bc993dcbe7099fd846"
Jul 06 21:07:44.669267 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:07:44.669248    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b60d9af4ce5dc72396be9c20f2d1d59eb9d4bf422c55bc993dcbe7099fd846\": container with ID starting with a1b60d9af4ce5dc72396be9c20f2d1d59eb9d4bf422c55bc993dcbe7099fd846 not found: ID does not exist" containerID="a1b60d9af4ce5dc72396be9c20f2d1d59eb9d4bf422c55bc993dcbe7099fd846"
Jul 06 21:07:44.669311 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.669273    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b60d9af4ce5dc72396be9c20f2d1d59eb9d4bf422c55bc993dcbe7099fd846"} err="failed to get container status \"a1b60d9af4ce5dc72396be9c20f2d1d59eb9d4bf422c55bc993dcbe7099fd846\": rpc error: code = NotFound desc = could not find container \"a1b60d9af4ce5dc72396be9c20f2d1d59eb9d4bf422c55bc993dcbe7099fd846\": container with ID starting with a1b60d9af4ce5dc72396be9c20f2d1d59eb9d4bf422c55bc993dcbe7099fd846 not found: ID does not exist"
Jul 06 21:07:44.682241 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.682215    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"]
Jul 06 21:07:44.701094 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:44.701067    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx"]
Jul 06 21:07:45.656472 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:07:45.656438    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40813f75-5b7b-497f-9477-0daa7dadfafc" path="/var/lib/kubelet/pods/40813f75-5b7b-497f-9477-0daa7dadfafc/volumes"
Jul 06 21:08:12.647766 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.647734    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"]
Jul 06 21:08:12.648158 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.648080    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40813f75-5b7b-497f-9477-0daa7dadfafc" containerName="storage-initializer"
Jul 06 21:08:12.648158 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.648092    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="40813f75-5b7b-497f-9477-0daa7dadfafc" containerName="storage-initializer"
Jul 06 21:08:12.648158 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.648109    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40813f75-5b7b-497f-9477-0daa7dadfafc" containerName="main"
Jul 06 21:08:12.648158 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.648115    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="40813f75-5b7b-497f-9477-0daa7dadfafc" containerName="main"
Jul 06 21:08:12.648158 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.648121    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caa807b9-8f0d-4996-905f-99ce948ac7ce" containerName="main"
Jul 06 21:08:12.648158 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.648127    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="caa807b9-8f0d-4996-905f-99ce948ac7ce" containerName="main"
Jul 06 21:08:12.648343 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.648179    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="40813f75-5b7b-497f-9477-0daa7dadfafc" containerName="main"
Jul 06 21:08:12.648343 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.648191    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="caa807b9-8f0d-4996-905f-99ce948ac7ce" containerName="main"
Jul 06 21:08:12.652169 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.652148    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:12.655184 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.655161    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:08:12.655329 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.655239    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:08:12.655897 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.655877    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g6qfk\""
Jul 06 21:08:12.655987 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.655913    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 06 21:08:12.666702 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.666681    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"]
Jul 06 21:08:12.689441 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.689409    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:12.689441 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.689444    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:12.689657 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.689489    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcf88798-5afe-4140-a59f-61cec023d362-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:12.689657 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.689531    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xhjx\" (UniqueName: \"kubernetes.io/projected/bcf88798-5afe-4140-a59f-61cec023d362-kube-api-access-9xhjx\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:12.689657 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.689567    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:12.689657 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.689585    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:12.790405 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.790367    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcf88798-5afe-4140-a59f-61cec023d362-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:12.790651 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.790416    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xhjx\" (UniqueName: \"kubernetes.io/projected/bcf88798-5afe-4140-a59f-61cec023d362-kube-api-access-9xhjx\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:12.790651 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.790456    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:12.790651 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.790475    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:12.790651 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.790510    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:12.790651 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.790530    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:12.790986 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.790967    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:12.791029 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.791010    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:12.791079 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.791044    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:12.792758 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.792728    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:12.793132 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.793112    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcf88798-5afe-4140-a59f-61cec023d362-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:12.802473 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.802446    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xhjx\" (UniqueName: \"kubernetes.io/projected/bcf88798-5afe-4140-a59f-61cec023d362-kube-api-access-9xhjx\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:12.962193 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:12.962105    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:13.098177 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:13.098152    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"]
Jul 06 21:08:13.100523 ip-10-0-134-202 kubenswrapper[2559]: W0706 21:08:13.100494    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf88798_5afe_4140_a59f_61cec023d362.slice/crio-bd4e5031c1ce3184ade9d0a98651d84bb8032a6d171a964787a2730d67e57866 WatchSource:0}: Error finding container bd4e5031c1ce3184ade9d0a98651d84bb8032a6d171a964787a2730d67e57866: Status 404 returned error can't find the container with id bd4e5031c1ce3184ade9d0a98651d84bb8032a6d171a964787a2730d67e57866
Jul 06 21:08:13.779950 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:13.779910    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf" event={"ID":"bcf88798-5afe-4140-a59f-61cec023d362","Type":"ContainerStarted","Data":"95b094760a19d6652f161f349391c12154a424de64cf663ee1ffdd03a4a71e2f"}
Jul 06 21:08:13.780406 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:13.779984    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf" event={"ID":"bcf88798-5afe-4140-a59f-61cec023d362","Type":"ContainerStarted","Data":"bd4e5031c1ce3184ade9d0a98651d84bb8032a6d171a964787a2730d67e57866"}
Jul 06 21:08:13.801765 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:13.801717    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf" podStartSLOduration=1.801702649 podStartE2EDuration="1.801702649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:08:13.800543273 +0000 UTC m=+1142.635430662" watchObservedRunningTime="2026-07-06 21:08:13.801702649 +0000 UTC m=+1142.636590014"
Jul 06 21:08:13.963088 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:13.963055    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:22.962246 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:22.962211    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:22.974777 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:22.974750    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:23.835402 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:23.835373    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:08:45.176315 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:45.176237    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"]
Jul 06 21:08:45.180319 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:45.180299    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"
Jul 06 21:08:45.182857 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:45.182826    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:08:45.182957 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:45.182889    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 06 21:08:45.183482 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:45.183462    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-epp-sa-dockercfg-wn82z\""
Jul 06 21:08:45.196246 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:45.196225    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"]
Jul 06 21:08:45.280575 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:45.280545    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/68d760b9-1e42-4ab9-b001-7aeb14096b04-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff\" (UID: \"68d760b9-1e42-4ab9-b001-7aeb14096b04\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"
Jul 06 21:08:45.280781 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:45.280594    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d760b9-1e42-4ab9-b001-7aeb14096b04-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff\" (UID: \"68d760b9-1e42-4ab9-b001-7aeb14096b04\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"
Jul 06 21:08:45.280781 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:45.280665    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68d760b9-1e42-4ab9-b001-7aeb14096b04-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff\" (UID: \"68d760b9-1e42-4ab9-b001-7aeb14096b04\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"
Jul 06 21:08:45.280781 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:45.280709    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lt7v\" (UniqueName: \"kubernetes.io/projected/68d760b9-1e42-4ab9-b001-7aeb14096b04-kube-api-access-7lt7v\") pod \"auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff\" (UID: \"68d760b9-1e42-4ab9-b001-7aeb14096b04\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"
Jul 06 21:08:45.381409 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:45.381375    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lt7v\" (UniqueName: \"kubernetes.io/projected/68d760b9-1e42-4ab9-b001-7aeb14096b04-kube-api-access-7lt7v\") pod \"auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff\" (UID: \"68d760b9-1e42-4ab9-b001-7aeb14096b04\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"
Jul 06 21:08:45.381577 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:45.381427    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/68d760b9-1e42-4ab9-b001-7aeb14096b04-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff\" (UID: \"68d760b9-1e42-4ab9-b001-7aeb14096b04\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"
Jul 06 21:08:45.381577 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:45.381459    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d760b9-1e42-4ab9-b001-7aeb14096b04-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff\" (UID: \"68d760b9-1e42-4ab9-b001-7aeb14096b04\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"
Jul 06 21:08:45.381577 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:45.381482    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68d760b9-1e42-4ab9-b001-7aeb14096b04-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff\" (UID: \"68d760b9-1e42-4ab9-b001-7aeb14096b04\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"
Jul 06 21:08:45.381849 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:45.381834    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68d760b9-1e42-4ab9-b001-7aeb14096b04-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff\" (UID: \"68d760b9-1e42-4ab9-b001-7aeb14096b04\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"
Jul 06 21:08:45.381890 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:45.381877    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/68d760b9-1e42-4ab9-b001-7aeb14096b04-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff\" (UID: \"68d760b9-1e42-4ab9-b001-7aeb14096b04\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"
Jul 06 21:08:45.383936 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:45.383917    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d760b9-1e42-4ab9-b001-7aeb14096b04-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff\" (UID: \"68d760b9-1e42-4ab9-b001-7aeb14096b04\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"
Jul 06 21:08:45.393409 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:45.393385    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lt7v\" (UniqueName: \"kubernetes.io/projected/68d760b9-1e42-4ab9-b001-7aeb14096b04-kube-api-access-7lt7v\") pod \"auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff\" (UID: \"68d760b9-1e42-4ab9-b001-7aeb14096b04\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"
Jul 06 21:08:45.490472 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:45.490392    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"
Jul 06 21:08:45.628284 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:45.628262    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"]
Jul 06 21:08:45.631044 ip-10-0-134-202 kubenswrapper[2559]: W0706 21:08:45.631018    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d760b9_1e42_4ab9_b001_7aeb14096b04.slice/crio-6acf8488a27f46824e9c251a0b7ea2babbd8936eef6063bcecf9968f2a7590a8 WatchSource:0}: Error finding container 6acf8488a27f46824e9c251a0b7ea2babbd8936eef6063bcecf9968f2a7590a8: Status 404 returned error can't find the container with id 6acf8488a27f46824e9c251a0b7ea2babbd8936eef6063bcecf9968f2a7590a8
Jul 06 21:08:45.924116 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:45.924080    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff" event={"ID":"68d760b9-1e42-4ab9-b001-7aeb14096b04","Type":"ContainerStarted","Data":"00a4ac9af085610bf9c4eb5bf0e711e5f9bfb8958470bc56820cf4fc2326c0e5"}
Jul 06 21:08:45.924288 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:45.924128    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff" event={"ID":"68d760b9-1e42-4ab9-b001-7aeb14096b04","Type":"ContainerStarted","Data":"6acf8488a27f46824e9c251a0b7ea2babbd8936eef6063bcecf9968f2a7590a8"}
Jul 06 21:08:46.929888 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:46.929853    2559 generic.go:356] "Generic (PLEG): container finished" podID="68d760b9-1e42-4ab9-b001-7aeb14096b04" containerID="00a4ac9af085610bf9c4eb5bf0e711e5f9bfb8958470bc56820cf4fc2326c0e5" exitCode=0
Jul 06 21:08:46.930308 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:46.929944    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff" event={"ID":"68d760b9-1e42-4ab9-b001-7aeb14096b04","Type":"ContainerDied","Data":"00a4ac9af085610bf9c4eb5bf0e711e5f9bfb8958470bc56820cf4fc2326c0e5"}
Jul 06 21:08:47.936834 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:47.936795    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff" event={"ID":"68d760b9-1e42-4ab9-b001-7aeb14096b04","Type":"ContainerStarted","Data":"bea32e814a6e80480fc069f82ccde314ba1eeedaad2b61201fd985bc99ce6ba2"}
Jul 06 21:08:47.937231 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:47.936903    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"
Jul 06 21:08:47.958835 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:08:47.958784    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff" podStartSLOduration=2.326220286 podStartE2EDuration="2.958769316s" totalImagesPullingTime="0s" totalInitContainerRuntime="632.54903ms" isStatefulPod=false podCreationTimestamp="2026-07-06 21:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:08:47.957649404 +0000 UTC m=+1176.792536790" watchObservedRunningTime="2026-07-06 21:08:47.958769316 +0000 UTC m=+1176.793656682"
Jul 06 21:09:18.943699 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:18.943671    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"
Jul 06 21:09:20.818773 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:20.818743    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"]
Jul 06 21:09:35.839040 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:35.838980    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf" podUID="bcf88798-5afe-4140-a59f-61cec023d362" containerName="main" containerID="cri-o://95b094760a19d6652f161f349391c12154a424de64cf663ee1ffdd03a4a71e2f" gracePeriod=45
Jul 06 21:09:36.087000 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.086977    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:09:36.154485 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.154406    2559 generic.go:356] "Generic (PLEG): container finished" podID="bcf88798-5afe-4140-a59f-61cec023d362" containerID="95b094760a19d6652f161f349391c12154a424de64cf663ee1ffdd03a4a71e2f" exitCode=0
Jul 06 21:09:36.154485 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.154454    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"
Jul 06 21:09:36.154829 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.154497    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf" event={"ID":"bcf88798-5afe-4140-a59f-61cec023d362","Type":"ContainerDied","Data":"95b094760a19d6652f161f349391c12154a424de64cf663ee1ffdd03a4a71e2f"}
Jul 06 21:09:36.154829 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.154559    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf" event={"ID":"bcf88798-5afe-4140-a59f-61cec023d362","Type":"ContainerDied","Data":"bd4e5031c1ce3184ade9d0a98651d84bb8032a6d171a964787a2730d67e57866"}
Jul 06 21:09:36.154829 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.154588    2559 scope.go:117] "RemoveContainer" containerID="95b094760a19d6652f161f349391c12154a424de64cf663ee1ffdd03a4a71e2f"
Jul 06 21:09:36.164568 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.164550    2559 scope.go:117] "RemoveContainer" containerID="95b094760a19d6652f161f349391c12154a424de64cf663ee1ffdd03a4a71e2f"
Jul 06 21:09:36.164842 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:09:36.164826    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b094760a19d6652f161f349391c12154a424de64cf663ee1ffdd03a4a71e2f\": container with ID starting with 95b094760a19d6652f161f349391c12154a424de64cf663ee1ffdd03a4a71e2f not found: ID does not exist" containerID="95b094760a19d6652f161f349391c12154a424de64cf663ee1ffdd03a4a71e2f"
Jul 06 21:09:36.164889 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.164849    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b094760a19d6652f161f349391c12154a424de64cf663ee1ffdd03a4a71e2f"} err="failed to get container status \"95b094760a19d6652f161f349391c12154a424de64cf663ee1ffdd03a4a71e2f\": rpc error: code = NotFound desc = could not find container \"95b094760a19d6652f161f349391c12154a424de64cf663ee1ffdd03a4a71e2f\": container with ID starting with 95b094760a19d6652f161f349391c12154a424de64cf663ee1ffdd03a4a71e2f not found: ID does not exist"
Jul 06 21:09:36.224534 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.224508    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcf88798-5afe-4140-a59f-61cec023d362-tls-certs\") pod \"bcf88798-5afe-4140-a59f-61cec023d362\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") "
Jul 06 21:09:36.224688 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.224541    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-tmp-dir\") pod \"bcf88798-5afe-4140-a59f-61cec023d362\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") "
Jul 06 21:09:36.224688 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.224558    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-model-cache\") pod \"bcf88798-5afe-4140-a59f-61cec023d362\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") "
Jul 06 21:09:36.224688 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.224599    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xhjx\" (UniqueName: \"kubernetes.io/projected/bcf88798-5afe-4140-a59f-61cec023d362-kube-api-access-9xhjx\") pod \"bcf88798-5afe-4140-a59f-61cec023d362\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") "
Jul 06 21:09:36.224688 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.224668    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-dshm\") pod \"bcf88798-5afe-4140-a59f-61cec023d362\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") "
Jul 06 21:09:36.224883 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.224692    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-home\") pod \"bcf88798-5afe-4140-a59f-61cec023d362\" (UID: \"bcf88798-5afe-4140-a59f-61cec023d362\") "
Jul 06 21:09:36.224883 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.224844    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-model-cache" (OuterVolumeSpecName: "model-cache") pod "bcf88798-5afe-4140-a59f-61cec023d362" (UID: "bcf88798-5afe-4140-a59f-61cec023d362"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:09:36.224883 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.224858    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bcf88798-5afe-4140-a59f-61cec023d362" (UID: "bcf88798-5afe-4140-a59f-61cec023d362"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:09:36.225045 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.225014    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-home" (OuterVolumeSpecName: "home") pod "bcf88798-5afe-4140-a59f-61cec023d362" (UID: "bcf88798-5afe-4140-a59f-61cec023d362"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:09:36.225097 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.225077    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-tmp-dir\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:09:36.225139 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.225096    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-model-cache\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:09:36.226770 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.226748    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-dshm" (OuterVolumeSpecName: "dshm") pod "bcf88798-5afe-4140-a59f-61cec023d362" (UID: "bcf88798-5afe-4140-a59f-61cec023d362"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:09:36.226850 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.226767    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf88798-5afe-4140-a59f-61cec023d362-kube-api-access-9xhjx" (OuterVolumeSpecName: "kube-api-access-9xhjx") pod "bcf88798-5afe-4140-a59f-61cec023d362" (UID: "bcf88798-5afe-4140-a59f-61cec023d362"). InnerVolumeSpecName "kube-api-access-9xhjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:09:36.226850 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.226806    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf88798-5afe-4140-a59f-61cec023d362-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bcf88798-5afe-4140-a59f-61cec023d362" (UID: "bcf88798-5afe-4140-a59f-61cec023d362"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:09:36.326484 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.326450    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcf88798-5afe-4140-a59f-61cec023d362-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:09:36.326484 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.326480    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xhjx\" (UniqueName: \"kubernetes.io/projected/bcf88798-5afe-4140-a59f-61cec023d362-kube-api-access-9xhjx\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:09:36.326484 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.326493    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-dshm\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:09:36.326748 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.326504    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcf88798-5afe-4140-a59f-61cec023d362-home\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:09:36.477392 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.477363    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"]
Jul 06 21:09:36.480682 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:36.480659    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf"]
Jul 06 21:09:37.657207 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:09:37.657174    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcf88798-5afe-4140-a59f-61cec023d362" path="/var/lib/kubelet/pods/bcf88798-5afe-4140-a59f-61cec023d362/volumes"
Jul 06 21:11:00.029825 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.029791    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"]
Jul 06 21:11:00.039856 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:11:00.039814    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:11:00.039856 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:11:00.039856 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 21:11:00.039856 ip-10-0-134-202 kubenswrapper[2559]:  > containerID="bea32e814a6e80480fc069f82ccde314ba1eeedaad2b61201fd985bc99ce6ba2" cmd=["/bin/sleep","15"]
Jul 06 21:11:00.040177 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.039866    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff" message="" err=<
Jul 06 21:11:00.040177 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:11:00.040177 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 21:11:00.040177 ip-10-0-134-202 kubenswrapper[2559]:  >
Jul 06 21:11:00.040177 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:11:00.039879    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:11:00.040177 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:11:00.040177 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 21:11:00.040177 ip-10-0-134-202 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff" podUID="68d760b9-1e42-4ab9-b001-7aeb14096b04" containerName="main" containerID="cri-o://bea32e814a6e80480fc069f82ccde314ba1eeedaad2b61201fd985bc99ce6ba2"
Jul 06 21:11:00.040177 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.039918    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff" podUID="68d760b9-1e42-4ab9-b001-7aeb14096b04" containerName="main" containerID="cri-o://bea32e814a6e80480fc069f82ccde314ba1eeedaad2b61201fd985bc99ce6ba2" gracePeriod=60
Jul 06 21:11:00.300316 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.300291    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"
Jul 06 21:11:00.344073 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.344030    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68d760b9-1e42-4ab9-b001-7aeb14096b04-kserve-provision-location\") pod \"68d760b9-1e42-4ab9-b001-7aeb14096b04\" (UID: \"68d760b9-1e42-4ab9-b001-7aeb14096b04\") "
Jul 06 21:11:00.344253 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.344091    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d760b9-1e42-4ab9-b001-7aeb14096b04-tls-certs\") pod \"68d760b9-1e42-4ab9-b001-7aeb14096b04\" (UID: \"68d760b9-1e42-4ab9-b001-7aeb14096b04\") "
Jul 06 21:11:00.344253 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.344118    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/68d760b9-1e42-4ab9-b001-7aeb14096b04-tokenizer-uds\") pod \"68d760b9-1e42-4ab9-b001-7aeb14096b04\" (UID: \"68d760b9-1e42-4ab9-b001-7aeb14096b04\") "
Jul 06 21:11:00.344253 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.344219    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lt7v\" (UniqueName: \"kubernetes.io/projected/68d760b9-1e42-4ab9-b001-7aeb14096b04-kube-api-access-7lt7v\") pod \"68d760b9-1e42-4ab9-b001-7aeb14096b04\" (UID: \"68d760b9-1e42-4ab9-b001-7aeb14096b04\") "
Jul 06 21:11:00.344436 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.344387    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d760b9-1e42-4ab9-b001-7aeb14096b04-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "68d760b9-1e42-4ab9-b001-7aeb14096b04" (UID: "68d760b9-1e42-4ab9-b001-7aeb14096b04"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:11:00.344789 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.344767    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d760b9-1e42-4ab9-b001-7aeb14096b04-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "68d760b9-1e42-4ab9-b001-7aeb14096b04" (UID: "68d760b9-1e42-4ab9-b001-7aeb14096b04"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:11:00.346183 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.346158    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68d760b9-1e42-4ab9-b001-7aeb14096b04-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68d760b9-1e42-4ab9-b001-7aeb14096b04" (UID: "68d760b9-1e42-4ab9-b001-7aeb14096b04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:11:00.346294 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.346194    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d760b9-1e42-4ab9-b001-7aeb14096b04-kube-api-access-7lt7v" (OuterVolumeSpecName: "kube-api-access-7lt7v") pod "68d760b9-1e42-4ab9-b001-7aeb14096b04" (UID: "68d760b9-1e42-4ab9-b001-7aeb14096b04"). InnerVolumeSpecName "kube-api-access-7lt7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:11:00.445273 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.445237    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lt7v\" (UniqueName: \"kubernetes.io/projected/68d760b9-1e42-4ab9-b001-7aeb14096b04-kube-api-access-7lt7v\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:11:00.445273 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.445266    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68d760b9-1e42-4ab9-b001-7aeb14096b04-kserve-provision-location\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:11:00.445273 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.445277    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d760b9-1e42-4ab9-b001-7aeb14096b04-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:11:00.445489 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.445286    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/68d760b9-1e42-4ab9-b001-7aeb14096b04-tokenizer-uds\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:11:00.542280 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.542193    2559 generic.go:356] "Generic (PLEG): container finished" podID="68d760b9-1e42-4ab9-b001-7aeb14096b04" containerID="bea32e814a6e80480fc069f82ccde314ba1eeedaad2b61201fd985bc99ce6ba2" exitCode=0
Jul 06 21:11:00.542280 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.542242    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"
Jul 06 21:11:00.542445 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.542281    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff" event={"ID":"68d760b9-1e42-4ab9-b001-7aeb14096b04","Type":"ContainerDied","Data":"bea32e814a6e80480fc069f82ccde314ba1eeedaad2b61201fd985bc99ce6ba2"}
Jul 06 21:11:00.542445 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.542355    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff" event={"ID":"68d760b9-1e42-4ab9-b001-7aeb14096b04","Type":"ContainerDied","Data":"6acf8488a27f46824e9c251a0b7ea2babbd8936eef6063bcecf9968f2a7590a8"}
Jul 06 21:11:00.542445 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.542376    2559 scope.go:117] "RemoveContainer" containerID="bea32e814a6e80480fc069f82ccde314ba1eeedaad2b61201fd985bc99ce6ba2"
Jul 06 21:11:00.551914 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.551896    2559 scope.go:117] "RemoveContainer" containerID="00a4ac9af085610bf9c4eb5bf0e711e5f9bfb8958470bc56820cf4fc2326c0e5"
Jul 06 21:11:00.559353 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.559334    2559 scope.go:117] "RemoveContainer" containerID="bea32e814a6e80480fc069f82ccde314ba1eeedaad2b61201fd985bc99ce6ba2"
Jul 06 21:11:00.559636 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:11:00.559602    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bea32e814a6e80480fc069f82ccde314ba1eeedaad2b61201fd985bc99ce6ba2\": container with ID starting with bea32e814a6e80480fc069f82ccde314ba1eeedaad2b61201fd985bc99ce6ba2 not found: ID does not exist" containerID="bea32e814a6e80480fc069f82ccde314ba1eeedaad2b61201fd985bc99ce6ba2"
Jul 06 21:11:00.559701 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.559648    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea32e814a6e80480fc069f82ccde314ba1eeedaad2b61201fd985bc99ce6ba2"} err="failed to get container status \"bea32e814a6e80480fc069f82ccde314ba1eeedaad2b61201fd985bc99ce6ba2\": rpc error: code = NotFound desc = could not find container \"bea32e814a6e80480fc069f82ccde314ba1eeedaad2b61201fd985bc99ce6ba2\": container with ID starting with bea32e814a6e80480fc069f82ccde314ba1eeedaad2b61201fd985bc99ce6ba2 not found: ID does not exist"
Jul 06 21:11:00.559701 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.559673    2559 scope.go:117] "RemoveContainer" containerID="00a4ac9af085610bf9c4eb5bf0e711e5f9bfb8958470bc56820cf4fc2326c0e5"
Jul 06 21:11:00.559931 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:11:00.559914    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00a4ac9af085610bf9c4eb5bf0e711e5f9bfb8958470bc56820cf4fc2326c0e5\": container with ID starting with 00a4ac9af085610bf9c4eb5bf0e711e5f9bfb8958470bc56820cf4fc2326c0e5 not found: ID does not exist" containerID="00a4ac9af085610bf9c4eb5bf0e711e5f9bfb8958470bc56820cf4fc2326c0e5"
Jul 06 21:11:00.559971 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.559938    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00a4ac9af085610bf9c4eb5bf0e711e5f9bfb8958470bc56820cf4fc2326c0e5"} err="failed to get container status \"00a4ac9af085610bf9c4eb5bf0e711e5f9bfb8958470bc56820cf4fc2326c0e5\": rpc error: code = NotFound desc = could not find container \"00a4ac9af085610bf9c4eb5bf0e711e5f9bfb8958470bc56820cf4fc2326c0e5\": container with ID starting with 00a4ac9af085610bf9c4eb5bf0e711e5f9bfb8958470bc56820cf4fc2326c0e5 not found: ID does not exist"
Jul 06 21:11:00.563494 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.563474    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"]
Jul 06 21:11:00.567430 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:00.567408    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff"]
Jul 06 21:11:01.657888 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:01.657860    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68d760b9-1e42-4ab9-b001-7aeb14096b04" path="/var/lib/kubelet/pods/68d760b9-1e42-4ab9-b001-7aeb14096b04/volumes"
Jul 06 21:11:20.683512 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.683479    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"]
Jul 06 21:11:20.683870 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.683839    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68d760b9-1e42-4ab9-b001-7aeb14096b04" containerName="main"
Jul 06 21:11:20.683870 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.683851    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="68d760b9-1e42-4ab9-b001-7aeb14096b04" containerName="main"
Jul 06 21:11:20.683870 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.683860    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68d760b9-1e42-4ab9-b001-7aeb14096b04" containerName="storage-initializer"
Jul 06 21:11:20.683870 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.683866    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="68d760b9-1e42-4ab9-b001-7aeb14096b04" containerName="storage-initializer"
Jul 06 21:11:20.683992 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.683874    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcf88798-5afe-4140-a59f-61cec023d362" containerName="main"
Jul 06 21:11:20.683992 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.683879    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcf88798-5afe-4140-a59f-61cec023d362" containerName="main"
Jul 06 21:11:20.683992 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.683937    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="68d760b9-1e42-4ab9-b001-7aeb14096b04" containerName="main"
Jul 06 21:11:20.683992 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.683947    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcf88798-5afe-4140-a59f-61cec023d362" containerName="main"
Jul 06 21:11:20.687908 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.687891    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.695707 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.695685    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:11:20.695795 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.695685    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 06 21:11:20.695795 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.695783    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-vdr6g\""
Jul 06 21:11:20.696346 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.696328    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:11:20.723835 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.723804    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"]
Jul 06 21:11:20.829591 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.829549    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90943f02-dd4a-435a-ae00-795cd937da15-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.829780 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.829596    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.829780 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.829706    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90943f02-dd4a-435a-ae00-795cd937da15-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.829780 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.829751    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.829780 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.829774    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90943f02-dd4a-435a-ae00-795cd937da15-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.829949 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.829850    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzpvm\" (UniqueName: \"kubernetes.io/projected/90943f02-dd4a-435a-ae00-795cd937da15-kube-api-access-rzpvm\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.829949 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.829886    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.829949 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.829903    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.829949 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.829934    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.930976 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.930890    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.930976 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.930930    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.930976 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.930955    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.931212 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.930985    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90943f02-dd4a-435a-ae00-795cd937da15-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.931212 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.931008    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.931212 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.931064    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90943f02-dd4a-435a-ae00-795cd937da15-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.931212 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.931113    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.931212 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.931134    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90943f02-dd4a-435a-ae00-795cd937da15-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.931212 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.931155    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzpvm\" (UniqueName: \"kubernetes.io/projected/90943f02-dd4a-435a-ae00-795cd937da15-kube-api-access-rzpvm\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.931481 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.931313    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.931481 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.931351    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.931481 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.931420    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.931685 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.931666    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90943f02-dd4a-435a-ae00-795cd937da15-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.931751 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.931718    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.933414 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.933392    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.933665 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.933649    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90943f02-dd4a-435a-ae00-795cd937da15-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.944944 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.944919    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90943f02-dd4a-435a-ae00-795cd937da15-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.945232 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.945212    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzpvm\" (UniqueName: \"kubernetes.io/projected/90943f02-dd4a-435a-ae00-795cd937da15-kube-api-access-rzpvm\") pod \"router-gateway-1-openshift-default-799f46c59b-pf9xz\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:20.998910 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:20.998876    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:21.138743 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:21.138716    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"]
Jul 06 21:11:21.140701 ip-10-0-134-202 kubenswrapper[2559]: W0706 21:11:21.140676    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90943f02_dd4a_435a_ae00_795cd937da15.slice/crio-800a7ee1fed5887062045705523c7256e567d88d1fa6e99e4370e04900dfa3aa WatchSource:0}: Error finding container 800a7ee1fed5887062045705523c7256e567d88d1fa6e99e4370e04900dfa3aa: Status 404 returned error can't find the container with id 800a7ee1fed5887062045705523c7256e567d88d1fa6e99e4370e04900dfa3aa
Jul 06 21:11:21.143390 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:21.143373    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 21:11:21.638602 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:21.638566    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz" event={"ID":"90943f02-dd4a-435a-ae00-795cd937da15","Type":"ContainerStarted","Data":"800a7ee1fed5887062045705523c7256e567d88d1fa6e99e4370e04900dfa3aa"}
Jul 06 21:11:23.785877 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:23.785841    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"]
Jul 06 21:11:26.805408 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:26.805372    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"]
Jul 06 21:11:26.809384 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:26.809360    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:26.812816 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:26.812790    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g6qfk\""
Jul 06 21:11:26.812941 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:26.812835    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Jul 06 21:11:26.825756 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:26.825690    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"]
Jul 06 21:11:26.986895 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:26.986860    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grfwf\" (UniqueName: \"kubernetes.io/projected/3416ec3a-a9c8-4005-bc4a-34598632f1cd-kube-api-access-grfwf\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:26.987093 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:26.986928    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:26.987093 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:26.986992    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:26.987093 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:26.987055    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:26.987225 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:26.987099    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:26.987225 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:26.987141    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3416ec3a-a9c8-4005-bc4a-34598632f1cd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:27.087758 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:27.087682    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:27.087758 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:27.087728    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:27.087758 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:27.087748    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:27.088031 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:27.087867    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3416ec3a-a9c8-4005-bc4a-34598632f1cd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:27.088031 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:27.087933    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grfwf\" (UniqueName: \"kubernetes.io/projected/3416ec3a-a9c8-4005-bc4a-34598632f1cd-kube-api-access-grfwf\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:27.088031 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:27.088004    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:27.088407 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:27.088227    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:27.088407 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:27.088360    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:27.088554 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:27.088411    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:27.090211 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:27.090188    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:27.090565 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:27.090542    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3416ec3a-a9c8-4005-bc4a-34598632f1cd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:27.096535 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:27.096512    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grfwf\" (UniqueName: \"kubernetes.io/projected/3416ec3a-a9c8-4005-bc4a-34598632f1cd-kube-api-access-grfwf\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:27.124115 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:27.124084    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:28.147585 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:28.147540    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:11:28.147965 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:28.147667    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:11:28.147965 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:28.147701    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:11:28.211009 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:28.210932    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"]
Jul 06 21:11:28.212926 ip-10-0-134-202 kubenswrapper[2559]: W0706 21:11:28.212901    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3416ec3a_a9c8_4005_bc4a_34598632f1cd.slice/crio-647dd8339a2d549175bb75b87c3dc89475045253755823706832f13d06a4b976 WatchSource:0}: Error finding container 647dd8339a2d549175bb75b87c3dc89475045253755823706832f13d06a4b976: Status 404 returned error can't find the container with id 647dd8339a2d549175bb75b87c3dc89475045253755823706832f13d06a4b976
Jul 06 21:11:28.674059 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:28.674022    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5" event={"ID":"3416ec3a-a9c8-4005-bc4a-34598632f1cd","Type":"ContainerStarted","Data":"f8c622c4cca52b65a0dfe832d15cc1a3cc0df33ce7232cde5aec858b51a20093"}
Jul 06 21:11:28.674262 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:28.674081    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5" event={"ID":"3416ec3a-a9c8-4005-bc4a-34598632f1cd","Type":"ContainerStarted","Data":"647dd8339a2d549175bb75b87c3dc89475045253755823706832f13d06a4b976"}
Jul 06 21:11:28.675471 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:28.675449    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz" event={"ID":"90943f02-dd4a-435a-ae00-795cd937da15","Type":"ContainerStarted","Data":"f7542cc665770b1d8cf4196e62f010699d92272418ba2d0ae80a076393fff8c9"}
Jul 06 21:11:28.675581 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:28.675534    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz" podUID="90943f02-dd4a-435a-ae00-795cd937da15" containerName="istio-proxy" containerID="cri-o://f7542cc665770b1d8cf4196e62f010699d92272418ba2d0ae80a076393fff8c9" gracePeriod=30
Jul 06 21:11:28.697594 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:28.697545    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5" podStartSLOduration=2.6975292 podStartE2EDuration="2.6975292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:11:28.696196132 +0000 UTC m=+1337.531083518" watchObservedRunningTime="2026-07-06 21:11:28.6975292 +0000 UTC m=+1337.532416566"
Jul 06 21:11:28.718329 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:28.718263    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz" podStartSLOduration=1.7145342989999999 podStartE2EDuration="8.718242523s" totalImagesPullingTime="7.003708224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:11:28.715957277 +0000 UTC m=+1337.550844669" watchObservedRunningTime="2026-07-06 21:11:28.718242523 +0000 UTC m=+1337.553129891"
Jul 06 21:11:29.124348 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:29.124299    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:35.999757 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:35.999673    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:37.124938 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:37.124845    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:37.137197 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:37.137165    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:37.727308 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:37.727280    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:11:39.629688 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.629657    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:39.726457 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.726380    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-credential-socket\") pod \"90943f02-dd4a-435a-ae00-795cd937da15\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") "
Jul 06 21:11:39.726457 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.726416    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-workload-certs\") pod \"90943f02-dd4a-435a-ae00-795cd937da15\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") "
Jul 06 21:11:39.726457 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.726445    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90943f02-dd4a-435a-ae00-795cd937da15-istiod-ca-cert\") pod \"90943f02-dd4a-435a-ae00-795cd937da15\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") "
Jul 06 21:11:39.726792 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.726473    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90943f02-dd4a-435a-ae00-795cd937da15-istio-token\") pod \"90943f02-dd4a-435a-ae00-795cd937da15\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") "
Jul 06 21:11:39.726792 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.726534    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-workload-socket\") pod \"90943f02-dd4a-435a-ae00-795cd937da15\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") "
Jul 06 21:11:39.726792 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.726565    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-istio-data\") pod \"90943f02-dd4a-435a-ae00-795cd937da15\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") "
Jul 06 21:11:39.726792 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.726646    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzpvm\" (UniqueName: \"kubernetes.io/projected/90943f02-dd4a-435a-ae00-795cd937da15-kube-api-access-rzpvm\") pod \"90943f02-dd4a-435a-ae00-795cd937da15\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") "
Jul 06 21:11:39.726792 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.726710    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90943f02-dd4a-435a-ae00-795cd937da15-istio-podinfo\") pod \"90943f02-dd4a-435a-ae00-795cd937da15\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") "
Jul 06 21:11:39.726792 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.726733    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-istio-envoy\") pod \"90943f02-dd4a-435a-ae00-795cd937da15\" (UID: \"90943f02-dd4a-435a-ae00-795cd937da15\") "
Jul 06 21:11:39.726792 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.726731    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "90943f02-dd4a-435a-ae00-795cd937da15" (UID: "90943f02-dd4a-435a-ae00-795cd937da15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:11:39.726792 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.726781    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "90943f02-dd4a-435a-ae00-795cd937da15" (UID: "90943f02-dd4a-435a-ae00-795cd937da15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:11:39.727155 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.726886    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-istio-data" (OuterVolumeSpecName: "istio-data") pod "90943f02-dd4a-435a-ae00-795cd937da15" (UID: "90943f02-dd4a-435a-ae00-795cd937da15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:11:39.727155 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.727000    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "90943f02-dd4a-435a-ae00-795cd937da15" (UID: "90943f02-dd4a-435a-ae00-795cd937da15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:11:39.727155 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.727048    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-credential-socket\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:11:39.727155 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.727069    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-workload-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:11:39.727155 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.727082    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-istio-data\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:11:39.727338 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.727199    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90943f02-dd4a-435a-ae00-795cd937da15-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "90943f02-dd4a-435a-ae00-795cd937da15" (UID: "90943f02-dd4a-435a-ae00-795cd937da15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:11:39.728469 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.728442    2559 generic.go:356] "Generic (PLEG): container finished" podID="90943f02-dd4a-435a-ae00-795cd937da15" containerID="f7542cc665770b1d8cf4196e62f010699d92272418ba2d0ae80a076393fff8c9" exitCode=0
Jul 06 21:11:39.728638 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.728500    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"
Jul 06 21:11:39.728638 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.728531    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz" event={"ID":"90943f02-dd4a-435a-ae00-795cd937da15","Type":"ContainerDied","Data":"f7542cc665770b1d8cf4196e62f010699d92272418ba2d0ae80a076393fff8c9"}
Jul 06 21:11:39.728638 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.728580    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz" event={"ID":"90943f02-dd4a-435a-ae00-795cd937da15","Type":"ContainerDied","Data":"800a7ee1fed5887062045705523c7256e567d88d1fa6e99e4370e04900dfa3aa"}
Jul 06 21:11:39.728638 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.728629    2559 scope.go:117] "RemoveContainer" containerID="f7542cc665770b1d8cf4196e62f010699d92272418ba2d0ae80a076393fff8c9"
Jul 06 21:11:39.729525 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.729501    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90943f02-dd4a-435a-ae00-795cd937da15-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "90943f02-dd4a-435a-ae00-795cd937da15" (UID: "90943f02-dd4a-435a-ae00-795cd937da15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 21:11:39.729649 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.729539    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90943f02-dd4a-435a-ae00-795cd937da15-kube-api-access-rzpvm" (OuterVolumeSpecName: "kube-api-access-rzpvm") pod "90943f02-dd4a-435a-ae00-795cd937da15" (UID: "90943f02-dd4a-435a-ae00-795cd937da15"). InnerVolumeSpecName "kube-api-access-rzpvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:11:39.729649 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.729558    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "90943f02-dd4a-435a-ae00-795cd937da15" (UID: "90943f02-dd4a-435a-ae00-795cd937da15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:11:39.729649 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.729640    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90943f02-dd4a-435a-ae00-795cd937da15-istio-token" (OuterVolumeSpecName: "istio-token") pod "90943f02-dd4a-435a-ae00-795cd937da15" (UID: "90943f02-dd4a-435a-ae00-795cd937da15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:11:39.747165 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.747145    2559 scope.go:117] "RemoveContainer" containerID="f7542cc665770b1d8cf4196e62f010699d92272418ba2d0ae80a076393fff8c9"
Jul 06 21:11:39.747432 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:11:39.747415    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7542cc665770b1d8cf4196e62f010699d92272418ba2d0ae80a076393fff8c9\": container with ID starting with f7542cc665770b1d8cf4196e62f010699d92272418ba2d0ae80a076393fff8c9 not found: ID does not exist" containerID="f7542cc665770b1d8cf4196e62f010699d92272418ba2d0ae80a076393fff8c9"
Jul 06 21:11:39.747491 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.747447    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7542cc665770b1d8cf4196e62f010699d92272418ba2d0ae80a076393fff8c9"} err="failed to get container status \"f7542cc665770b1d8cf4196e62f010699d92272418ba2d0ae80a076393fff8c9\": rpc error: code = NotFound desc = could not find container \"f7542cc665770b1d8cf4196e62f010699d92272418ba2d0ae80a076393fff8c9\": container with ID starting with f7542cc665770b1d8cf4196e62f010699d92272418ba2d0ae80a076393fff8c9 not found: ID does not exist"
Jul 06 21:11:39.828067 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.828028    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzpvm\" (UniqueName: \"kubernetes.io/projected/90943f02-dd4a-435a-ae00-795cd937da15-kube-api-access-rzpvm\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:11:39.828067 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.828061    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90943f02-dd4a-435a-ae00-795cd937da15-istio-podinfo\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:11:39.828067 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.828073    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-istio-envoy\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:11:39.828356 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.828083    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90943f02-dd4a-435a-ae00-795cd937da15-istiod-ca-cert\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:11:39.828356 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.828095    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90943f02-dd4a-435a-ae00-795cd937da15-istio-token\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:11:39.828356 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:39.828106    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90943f02-dd4a-435a-ae00-795cd937da15-workload-socket\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:11:40.061450 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:40.061415    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"]
Jul 06 21:11:40.071260 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:40.071231    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz"]
Jul 06 21:11:41.657765 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:11:41.657730    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90943f02-dd4a-435a-ae00-795cd937da15" path="/var/lib/kubelet/pods/90943f02-dd4a-435a-ae00-795cd937da15/volumes"
Jul 06 21:13:50.454677 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:13:50.454645    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"]
Jul 06 21:14:05.474850 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:05.474804    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5" podUID="3416ec3a-a9c8-4005-bc4a-34598632f1cd" containerName="main" containerID="cri-o://f8c622c4cca52b65a0dfe832d15cc1a3cc0df33ce7232cde5aec858b51a20093" gracePeriod=45
Jul 06 21:14:05.725277 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:05.725212    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:14:05.823237 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:05.823202    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grfwf\" (UniqueName: \"kubernetes.io/projected/3416ec3a-a9c8-4005-bc4a-34598632f1cd-kube-api-access-grfwf\") pod \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") "
Jul 06 21:14:05.823407 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:05.823272    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-model-cache\") pod \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") "
Jul 06 21:14:05.823407 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:05.823326    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-home\") pod \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") "
Jul 06 21:14:05.823407 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:05.823341    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3416ec3a-a9c8-4005-bc4a-34598632f1cd-tls-certs\") pod \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") "
Jul 06 21:14:05.823407 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:05.823384    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-dshm\") pod \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") "
Jul 06 21:14:05.823407 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:05.823405    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-tmp-dir\") pod \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\" (UID: \"3416ec3a-a9c8-4005-bc4a-34598632f1cd\") "
Jul 06 21:14:05.823707 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:05.823537    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "3416ec3a-a9c8-4005-bc4a-34598632f1cd" (UID: "3416ec3a-a9c8-4005-bc4a-34598632f1cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:14:05.823707 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:05.823549    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-home" (OuterVolumeSpecName: "home") pod "3416ec3a-a9c8-4005-bc4a-34598632f1cd" (UID: "3416ec3a-a9c8-4005-bc4a-34598632f1cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:14:05.823861 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:05.823752    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3416ec3a-a9c8-4005-bc4a-34598632f1cd" (UID: "3416ec3a-a9c8-4005-bc4a-34598632f1cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:14:05.825410 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:05.825383    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-dshm" (OuterVolumeSpecName: "dshm") pod "3416ec3a-a9c8-4005-bc4a-34598632f1cd" (UID: "3416ec3a-a9c8-4005-bc4a-34598632f1cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:14:05.825520 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:05.825406    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3416ec3a-a9c8-4005-bc4a-34598632f1cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3416ec3a-a9c8-4005-bc4a-34598632f1cd" (UID: "3416ec3a-a9c8-4005-bc4a-34598632f1cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:14:05.825520 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:05.825393    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3416ec3a-a9c8-4005-bc4a-34598632f1cd-kube-api-access-grfwf" (OuterVolumeSpecName: "kube-api-access-grfwf") pod "3416ec3a-a9c8-4005-bc4a-34598632f1cd" (UID: "3416ec3a-a9c8-4005-bc4a-34598632f1cd"). InnerVolumeSpecName "kube-api-access-grfwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:14:05.924010 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:05.923975    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-dshm\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:14:05.924231 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:05.924026    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-tmp-dir\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:14:05.924231 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:05.924038    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grfwf\" (UniqueName: \"kubernetes.io/projected/3416ec3a-a9c8-4005-bc4a-34598632f1cd-kube-api-access-grfwf\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:14:05.924231 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:05.924049    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-model-cache\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:14:05.924231 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:05.924057    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3416ec3a-a9c8-4005-bc4a-34598632f1cd-home\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:14:05.924231 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:05.924065    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3416ec3a-a9c8-4005-bc4a-34598632f1cd-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:14:06.402716 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:06.402683    2559 generic.go:356] "Generic (PLEG): container finished" podID="3416ec3a-a9c8-4005-bc4a-34598632f1cd" containerID="f8c622c4cca52b65a0dfe832d15cc1a3cc0df33ce7232cde5aec858b51a20093" exitCode=0
Jul 06 21:14:06.402914 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:06.402737    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5" event={"ID":"3416ec3a-a9c8-4005-bc4a-34598632f1cd","Type":"ContainerDied","Data":"f8c622c4cca52b65a0dfe832d15cc1a3cc0df33ce7232cde5aec858b51a20093"}
Jul 06 21:14:06.402914 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:06.402743    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"
Jul 06 21:14:06.402914 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:06.402776    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5" event={"ID":"3416ec3a-a9c8-4005-bc4a-34598632f1cd","Type":"ContainerDied","Data":"647dd8339a2d549175bb75b87c3dc89475045253755823706832f13d06a4b976"}
Jul 06 21:14:06.402914 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:06.402797    2559 scope.go:117] "RemoveContainer" containerID="f8c622c4cca52b65a0dfe832d15cc1a3cc0df33ce7232cde5aec858b51a20093"
Jul 06 21:14:06.411670 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:06.411653    2559 scope.go:117] "RemoveContainer" containerID="f8c622c4cca52b65a0dfe832d15cc1a3cc0df33ce7232cde5aec858b51a20093"
Jul 06 21:14:06.411919 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:14:06.411902    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c622c4cca52b65a0dfe832d15cc1a3cc0df33ce7232cde5aec858b51a20093\": container with ID starting with f8c622c4cca52b65a0dfe832d15cc1a3cc0df33ce7232cde5aec858b51a20093 not found: ID does not exist" containerID="f8c622c4cca52b65a0dfe832d15cc1a3cc0df33ce7232cde5aec858b51a20093"
Jul 06 21:14:06.411983 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:06.411934    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c622c4cca52b65a0dfe832d15cc1a3cc0df33ce7232cde5aec858b51a20093"} err="failed to get container status \"f8c622c4cca52b65a0dfe832d15cc1a3cc0df33ce7232cde5aec858b51a20093\": rpc error: code = NotFound desc = could not find container \"f8c622c4cca52b65a0dfe832d15cc1a3cc0df33ce7232cde5aec858b51a20093\": container with ID starting with f8c622c4cca52b65a0dfe832d15cc1a3cc0df33ce7232cde5aec858b51a20093 not found: ID does not exist"
Jul 06 21:14:06.424139 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:06.424115    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"]
Jul 06 21:14:06.428943 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:06.428923    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5"]
Jul 06 21:14:06.717777 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:06.717673    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5" podUID="3416ec3a-a9c8-4005-bc4a-34598632f1cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": context deadline exceeded"
Jul 06 21:14:07.657321 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:07.657285    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3416ec3a-a9c8-4005-bc4a-34598632f1cd" path="/var/lib/kubelet/pods/3416ec3a-a9c8-4005-bc4a-34598632f1cd/volumes"
Jul 06 21:14:30.544363 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:30.544330    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-z62nj"]
Jul 06 21:14:30.544852 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:30.544680    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90943f02-dd4a-435a-ae00-795cd937da15" containerName="istio-proxy"
Jul 06 21:14:30.544852 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:30.544691    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="90943f02-dd4a-435a-ae00-795cd937da15" containerName="istio-proxy"
Jul 06 21:14:30.544852 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:30.544704    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3416ec3a-a9c8-4005-bc4a-34598632f1cd" containerName="main"
Jul 06 21:14:30.544852 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:30.544709    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3416ec3a-a9c8-4005-bc4a-34598632f1cd" containerName="main"
Jul 06 21:14:30.544852 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:30.544768    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="90943f02-dd4a-435a-ae00-795cd937da15" containerName="istio-proxy"
Jul 06 21:14:30.544852 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:30.544775    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3416ec3a-a9c8-4005-bc4a-34598632f1cd" containerName="main"
Jul 06 21:14:30.548749 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:30.548734    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-z62nj"
Jul 06 21:14:30.551125 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:30.551105    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:14:30.551851 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:30.551832    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g6qfk\""
Jul 06 21:14:30.551851 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:30.551832    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:14:30.555685 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:30.555405    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-z62nj"]
Jul 06 21:14:30.644518 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:30.644482    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dc4z\" (UniqueName: \"kubernetes.io/projected/434956f2-26d1-4e99-8e60-363e99f00b0e-kube-api-access-5dc4z\") pod \"e2e-pvc-model-download-z62nj\" (UID: \"434956f2-26d1-4e99-8e60-363e99f00b0e\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-z62nj"
Jul 06 21:14:30.745264 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:30.745235    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dc4z\" (UniqueName: \"kubernetes.io/projected/434956f2-26d1-4e99-8e60-363e99f00b0e-kube-api-access-5dc4z\") pod \"e2e-pvc-model-download-z62nj\" (UID: \"434956f2-26d1-4e99-8e60-363e99f00b0e\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-z62nj"
Jul 06 21:14:30.753410 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:30.753387    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dc4z\" (UniqueName: \"kubernetes.io/projected/434956f2-26d1-4e99-8e60-363e99f00b0e-kube-api-access-5dc4z\") pod \"e2e-pvc-model-download-z62nj\" (UID: \"434956f2-26d1-4e99-8e60-363e99f00b0e\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-z62nj"
Jul 06 21:14:35.485977 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:35.485884    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"e2e-pvc-model-download-z62nj\" (UID: \"434956f2-26d1-4e99-8e60-363e99f00b0e\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-z62nj"
Jul 06 21:14:35.495274 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:35.495246    2559 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"e2e-pvc-model-download-z62nj\" (UID: \"434956f2-26d1-4e99-8e60-363e99f00b0e\") device path: \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-z62nj"
Jul 06 21:14:35.587010 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:35.586976    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"e2e-pvc-model-download-z62nj\" (UID: \"434956f2-26d1-4e99-8e60-363e99f00b0e\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-z62nj"
Jul 06 21:14:35.587186 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:35.587128    2559 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"e2e-pvc-model-download-z62nj\" (UID: \"434956f2-26d1-4e99-8e60-363e99f00b0e\") DevicePath \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-z62nj"
Jul 06 21:14:35.590874 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:35.590852    2559 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"e2e-pvc-model-download-z62nj\" (UID: \"434956f2-26d1-4e99-8e60-363e99f00b0e\") DevicePath \"csi-7af66be380d3cfadc82c49ce204ebedd23f1b6b197ce54aeef71486c0a6c30d9\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-z62nj"
Jul 06 21:14:35.807916 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:35.807882    2559 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"e2e-pvc-model-download-z62nj\" (UID: \"434956f2-26d1-4e99-8e60-363e99f00b0e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/695b1fedd1efd2a2620b46c540e3c4cbf5ecfc1b083102e8f2c618a1c9ec9ff5/globalmount\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-z62nj"
Jul 06 21:14:35.825424 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:35.825394    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"e2e-pvc-model-download-z62nj\" (UID: \"434956f2-26d1-4e99-8e60-363e99f00b0e\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-z62nj"
Jul 06 21:14:35.957914 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:35.957887    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-z62nj"
Jul 06 21:14:36.096705 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:36.096679    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-z62nj"]
Jul 06 21:14:36.099232 ip-10-0-134-202 kubenswrapper[2559]: W0706 21:14:36.099201    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod434956f2_26d1_4e99_8e60_363e99f00b0e.slice/crio-f640eb49f4ef82e9fb5c0a0608959089294fe48607931a198745c36967f42ebe WatchSource:0}: Error finding container f640eb49f4ef82e9fb5c0a0608959089294fe48607931a198745c36967f42ebe: Status 404 returned error can't find the container with id f640eb49f4ef82e9fb5c0a0608959089294fe48607931a198745c36967f42ebe
Jul 06 21:14:36.541714 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:36.541681    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-z62nj" event={"ID":"434956f2-26d1-4e99-8e60-363e99f00b0e","Type":"ContainerStarted","Data":"4bcd725c95d2c174f9932569851cdabd50f1af48e8d113e178ced4c5015b29f2"}
Jul 06 21:14:36.542072 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:36.541722    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-z62nj" event={"ID":"434956f2-26d1-4e99-8e60-363e99f00b0e","Type":"ContainerStarted","Data":"f640eb49f4ef82e9fb5c0a0608959089294fe48607931a198745c36967f42ebe"}
Jul 06 21:14:36.562211 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:36.562161    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/e2e-pvc-model-download-z62nj" podStartSLOduration=9.562144913000001 podStartE2EDuration="9.562144913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:14:36.559366881 +0000 UTC m=+1525.394254247" watchObservedRunningTime="2026-07-06 21:14:36.562144913 +0000 UTC m=+1525.397032281"
Jul 06 21:14:42.570870 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:42.570840    2559 generic.go:356] "Generic (PLEG): container finished" podID="434956f2-26d1-4e99-8e60-363e99f00b0e" containerID="4bcd725c95d2c174f9932569851cdabd50f1af48e8d113e178ced4c5015b29f2" exitCode=0
Jul 06 21:14:42.571273 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:42.570912    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-z62nj" event={"ID":"434956f2-26d1-4e99-8e60-363e99f00b0e","Type":"ContainerDied","Data":"4bcd725c95d2c174f9932569851cdabd50f1af48e8d113e178ced4c5015b29f2"}
Jul 06 21:14:43.720928 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:43.720905    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-z62nj"
Jul 06 21:14:43.860928 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:43.860844    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") pod \"434956f2-26d1-4e99-8e60-363e99f00b0e\" (UID: \"434956f2-26d1-4e99-8e60-363e99f00b0e\") "
Jul 06 21:14:43.860928 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:43.860892    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dc4z\" (UniqueName: \"kubernetes.io/projected/434956f2-26d1-4e99-8e60-363e99f00b0e-kube-api-access-5dc4z\") pod \"434956f2-26d1-4e99-8e60-363e99f00b0e\" (UID: \"434956f2-26d1-4e99-8e60-363e99f00b0e\") "
Jul 06 21:14:43.863045 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:43.863020    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434956f2-26d1-4e99-8e60-363e99f00b0e-kube-api-access-5dc4z" (OuterVolumeSpecName: "kube-api-access-5dc4z") pod "434956f2-26d1-4e99-8e60-363e99f00b0e" (UID: "434956f2-26d1-4e99-8e60-363e99f00b0e"). InnerVolumeSpecName "kube-api-access-5dc4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:14:43.864506 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:43.864486    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e" (OuterVolumeSpecName: "model-storage") pod "434956f2-26d1-4e99-8e60-363e99f00b0e" (UID: "434956f2-26d1-4e99-8e60-363e99f00b0e"). InnerVolumeSpecName "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 06 21:14:43.961699 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:43.961668    2559 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") on node \"ip-10-0-134-202.ec2.internal\" "
Jul 06 21:14:43.961699 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:43.961699    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dc4z\" (UniqueName: \"kubernetes.io/projected/434956f2-26d1-4e99-8e60-363e99f00b0e-kube-api-access-5dc4z\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:14:44.073058 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:44.073040    2559 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e") on node "ip-10-0-134-202.ec2.internal"
Jul 06 21:14:44.163772 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:44.163689    2559 reconciler_common.go:299] "Volume detached for volume \"pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-000d63f5e0662418e\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"csi-7af66be380d3cfadc82c49ce204ebedd23f1b6b197ce54aeef71486c0a6c30d9\""
Jul 06 21:14:44.581412 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:44.581376    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-z62nj" event={"ID":"434956f2-26d1-4e99-8e60-363e99f00b0e","Type":"ContainerDied","Data":"f640eb49f4ef82e9fb5c0a0608959089294fe48607931a198745c36967f42ebe"}
Jul 06 21:14:44.581412 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:44.581393    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-z62nj"
Jul 06 21:14:44.581412 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:44.581414    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f640eb49f4ef82e9fb5c0a0608959089294fe48607931a198745c36967f42ebe"
Jul 06 21:14:56.730365 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:56.730322    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q"]
Jul 06 21:14:56.730904 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:56.730882    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="434956f2-26d1-4e99-8e60-363e99f00b0e" containerName="storage-initializer"
Jul 06 21:14:56.730952 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:56.730908    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="434956f2-26d1-4e99-8e60-363e99f00b0e" containerName="storage-initializer"
Jul 06 21:14:56.731051 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:56.731039    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="434956f2-26d1-4e99-8e60-363e99f00b0e" containerName="storage-initializer"
Jul 06 21:14:56.734429 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:56.734408    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q"
Jul 06 21:14:56.739252 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:56.739225    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:14:56.739382 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:56.739261    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-z82dw\""
Jul 06 21:14:56.739382 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:56.739362    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g6qfk\""
Jul 06 21:14:56.739505 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:56.739488    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 06 21:14:56.739566 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:56.739512    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:14:56.747960 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:56.747936    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q"]
Jul 06 21:14:56.773275 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:56.773245    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb5643f2-9ad6-46f6-ba2f-2e8a23a05500-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q\" (UID: \"bb5643f2-9ad6-46f6-ba2f-2e8a23a05500\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q"
Jul 06 21:14:56.773423 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:56.773309    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5nwb\" (UniqueName: \"kubernetes.io/projected/bb5643f2-9ad6-46f6-ba2f-2e8a23a05500-kube-api-access-b5nwb\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q\" (UID: \"bb5643f2-9ad6-46f6-ba2f-2e8a23a05500\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q"
Jul 06 21:14:56.773423 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:56.773360    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bb5643f2-9ad6-46f6-ba2f-2e8a23a05500-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q\" (UID: \"bb5643f2-9ad6-46f6-ba2f-2e8a23a05500\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q"
Jul 06 21:14:56.874409 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:56.874373    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bb5643f2-9ad6-46f6-ba2f-2e8a23a05500-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q\" (UID: \"bb5643f2-9ad6-46f6-ba2f-2e8a23a05500\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q"
Jul 06 21:14:56.874580 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:56.874425    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb5643f2-9ad6-46f6-ba2f-2e8a23a05500-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q\" (UID: \"bb5643f2-9ad6-46f6-ba2f-2e8a23a05500\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q"
Jul 06 21:14:56.874580 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:56.874464    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5nwb\" (UniqueName: \"kubernetes.io/projected/bb5643f2-9ad6-46f6-ba2f-2e8a23a05500-kube-api-access-b5nwb\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q\" (UID: \"bb5643f2-9ad6-46f6-ba2f-2e8a23a05500\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q"
Jul 06 21:14:56.874868 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:56.874846    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bb5643f2-9ad6-46f6-ba2f-2e8a23a05500-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q\" (UID: \"bb5643f2-9ad6-46f6-ba2f-2e8a23a05500\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q"
Jul 06 21:14:56.876863 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:56.876845    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb5643f2-9ad6-46f6-ba2f-2e8a23a05500-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q\" (UID: \"bb5643f2-9ad6-46f6-ba2f-2e8a23a05500\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q"
Jul 06 21:14:56.884549 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:56.884523    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5nwb\" (UniqueName: \"kubernetes.io/projected/bb5643f2-9ad6-46f6-ba2f-2e8a23a05500-kube-api-access-b5nwb\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q\" (UID: \"bb5643f2-9ad6-46f6-ba2f-2e8a23a05500\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q"
Jul 06 21:14:57.044191 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:57.044157    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q"
Jul 06 21:14:57.178394 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:57.178373    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q"]
Jul 06 21:14:57.180420 ip-10-0-134-202 kubenswrapper[2559]: W0706 21:14:57.180390    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5643f2_9ad6_46f6_ba2f_2e8a23a05500.slice/crio-811a5420059ee7accf598e2ba88d8993a43e5688dd4775623e1e52f58121791f WatchSource:0}: Error finding container 811a5420059ee7accf598e2ba88d8993a43e5688dd4775623e1e52f58121791f: Status 404 returned error can't find the container with id 811a5420059ee7accf598e2ba88d8993a43e5688dd4775623e1e52f58121791f
Jul 06 21:14:57.643664 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:57.643600    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q" event={"ID":"bb5643f2-9ad6-46f6-ba2f-2e8a23a05500","Type":"ContainerStarted","Data":"0ba6b709361802a8991018647b0a58b0c26cb3f615e66ef5d40353c3ebc8ab42"}
Jul 06 21:14:57.643870 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:57.643705    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q" event={"ID":"bb5643f2-9ad6-46f6-ba2f-2e8a23a05500","Type":"ContainerStarted","Data":"811a5420059ee7accf598e2ba88d8993a43e5688dd4775623e1e52f58121791f"}
Jul 06 21:14:57.643870 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:57.643730    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q"
Jul 06 21:14:57.665492 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:14:57.665445    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q" podStartSLOduration=1.665430759 podStartE2EDuration="1.665430759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:14:57.663833177 +0000 UTC m=+1546.498720542" watchObservedRunningTime="2026-07-06 21:14:57.665430759 +0000 UTC m=+1546.500318126"
Jul 06 21:15:28.649868 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:15:28.649838    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q"
Jul 06 21:17:51.057624 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.057510    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q"]
Jul 06 21:17:51.066184 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:17:51.066154    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:17:51.066184 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:17:51.066184 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 21:17:51.066184 ip-10-0-134-202 kubenswrapper[2559]:  > containerID="0ba6b709361802a8991018647b0a58b0c26cb3f615e66ef5d40353c3ebc8ab42" cmd=["/bin/sleep","15"]
Jul 06 21:17:51.066409 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.066202    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q" message="" err=<
Jul 06 21:17:51.066409 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:17:51.066409 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 21:17:51.066409 ip-10-0-134-202 kubenswrapper[2559]:  >
Jul 06 21:17:51.066409 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:17:51.066214    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:17:51.066409 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:17:51.066409 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 21:17:51.066409 ip-10-0-134-202 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q" podUID="bb5643f2-9ad6-46f6-ba2f-2e8a23a05500" containerName="main" containerID="cri-o://0ba6b709361802a8991018647b0a58b0c26cb3f615e66ef5d40353c3ebc8ab42"
Jul 06 21:17:51.066409 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.066259    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q" podUID="bb5643f2-9ad6-46f6-ba2f-2e8a23a05500" containerName="main" containerID="cri-o://0ba6b709361802a8991018647b0a58b0c26cb3f615e66ef5d40353c3ebc8ab42" gracePeriod=60
Jul 06 21:17:51.314714 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.314559    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q"
Jul 06 21:17:51.399708 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.399683    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5nwb\" (UniqueName: \"kubernetes.io/projected/bb5643f2-9ad6-46f6-ba2f-2e8a23a05500-kube-api-access-b5nwb\") pod \"bb5643f2-9ad6-46f6-ba2f-2e8a23a05500\" (UID: \"bb5643f2-9ad6-46f6-ba2f-2e8a23a05500\") "
Jul 06 21:17:51.399828 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.399727    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bb5643f2-9ad6-46f6-ba2f-2e8a23a05500-tokenizer-uds\") pod \"bb5643f2-9ad6-46f6-ba2f-2e8a23a05500\" (UID: \"bb5643f2-9ad6-46f6-ba2f-2e8a23a05500\") "
Jul 06 21:17:51.399828 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.399747    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb5643f2-9ad6-46f6-ba2f-2e8a23a05500-tls-certs\") pod \"bb5643f2-9ad6-46f6-ba2f-2e8a23a05500\" (UID: \"bb5643f2-9ad6-46f6-ba2f-2e8a23a05500\") "
Jul 06 21:17:51.399958 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.399938    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5643f2-9ad6-46f6-ba2f-2e8a23a05500-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "bb5643f2-9ad6-46f6-ba2f-2e8a23a05500" (UID: "bb5643f2-9ad6-46f6-ba2f-2e8a23a05500"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:17:51.401666 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.401637    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5643f2-9ad6-46f6-ba2f-2e8a23a05500-kube-api-access-b5nwb" (OuterVolumeSpecName: "kube-api-access-b5nwb") pod "bb5643f2-9ad6-46f6-ba2f-2e8a23a05500" (UID: "bb5643f2-9ad6-46f6-ba2f-2e8a23a05500"). InnerVolumeSpecName "kube-api-access-b5nwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:17:51.401765 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.401695    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5643f2-9ad6-46f6-ba2f-2e8a23a05500-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb5643f2-9ad6-46f6-ba2f-2e8a23a05500" (UID: "bb5643f2-9ad6-46f6-ba2f-2e8a23a05500"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:17:51.435661 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.435641    2559 generic.go:356] "Generic (PLEG): container finished" podID="bb5643f2-9ad6-46f6-ba2f-2e8a23a05500" containerID="0ba6b709361802a8991018647b0a58b0c26cb3f615e66ef5d40353c3ebc8ab42" exitCode=0
Jul 06 21:17:51.435761 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.435685    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q"
Jul 06 21:17:51.435761 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.435721    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q" event={"ID":"bb5643f2-9ad6-46f6-ba2f-2e8a23a05500","Type":"ContainerDied","Data":"0ba6b709361802a8991018647b0a58b0c26cb3f615e66ef5d40353c3ebc8ab42"}
Jul 06 21:17:51.435761 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.435754    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q" event={"ID":"bb5643f2-9ad6-46f6-ba2f-2e8a23a05500","Type":"ContainerDied","Data":"811a5420059ee7accf598e2ba88d8993a43e5688dd4775623e1e52f58121791f"}
Jul 06 21:17:51.435867 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.435774    2559 scope.go:117] "RemoveContainer" containerID="0ba6b709361802a8991018647b0a58b0c26cb3f615e66ef5d40353c3ebc8ab42"
Jul 06 21:17:51.444266 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.444251    2559 scope.go:117] "RemoveContainer" containerID="0ba6b709361802a8991018647b0a58b0c26cb3f615e66ef5d40353c3ebc8ab42"
Jul 06 21:17:51.444516 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:17:51.444494    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ba6b709361802a8991018647b0a58b0c26cb3f615e66ef5d40353c3ebc8ab42\": container with ID starting with 0ba6b709361802a8991018647b0a58b0c26cb3f615e66ef5d40353c3ebc8ab42 not found: ID does not exist" containerID="0ba6b709361802a8991018647b0a58b0c26cb3f615e66ef5d40353c3ebc8ab42"
Jul 06 21:17:51.444594 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.444526    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba6b709361802a8991018647b0a58b0c26cb3f615e66ef5d40353c3ebc8ab42"} err="failed to get container status \"0ba6b709361802a8991018647b0a58b0c26cb3f615e66ef5d40353c3ebc8ab42\": rpc error: code = NotFound desc = could not find container \"0ba6b709361802a8991018647b0a58b0c26cb3f615e66ef5d40353c3ebc8ab42\": container with ID starting with 0ba6b709361802a8991018647b0a58b0c26cb3f615e66ef5d40353c3ebc8ab42 not found: ID does not exist"
Jul 06 21:17:51.458292 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.458271    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q"]
Jul 06 21:17:51.465306 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.465288    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q"]
Jul 06 21:17:51.501133 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.501111    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5nwb\" (UniqueName: \"kubernetes.io/projected/bb5643f2-9ad6-46f6-ba2f-2e8a23a05500-kube-api-access-b5nwb\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:17:51.501133 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.501128    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bb5643f2-9ad6-46f6-ba2f-2e8a23a05500-tokenizer-uds\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:17:51.501250 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.501138    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb5643f2-9ad6-46f6-ba2f-2e8a23a05500-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:17:51.657522 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:17:51.657451    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb5643f2-9ad6-46f6-ba2f-2e8a23a05500" path="/var/lib/kubelet/pods/bb5643f2-9ad6-46f6-ba2f-2e8a23a05500/volumes"
Jul 06 21:18:18.131030 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:18.130989    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk"]
Jul 06 21:18:18.131363 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:18.131327    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb5643f2-9ad6-46f6-ba2f-2e8a23a05500" containerName="main"
Jul 06 21:18:18.131363 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:18.131339    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb5643f2-9ad6-46f6-ba2f-2e8a23a05500" containerName="main"
Jul 06 21:18:18.131444 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:18.131410    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb5643f2-9ad6-46f6-ba2f-2e8a23a05500" containerName="main"
Jul 06 21:18:18.134274 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:18.134259    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk"
Jul 06 21:18:18.136779 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:18.136758    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 06 21:18:18.137445 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:18.137425    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g6qfk\""
Jul 06 21:18:18.137533 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:18.137463    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:18:18.137533 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:18.137516    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-ch2j2\""
Jul 06 21:18:18.137662 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:18.137647    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:18:18.146872 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:18.146853    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk"]
Jul 06 21:18:18.219031 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:18.218999    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpkvm\" (UniqueName: \"kubernetes.io/projected/bf3f1f43-f34e-4fec-b22e-7abe0484d800-kube-api-access-vpkvm\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk\" (UID: \"bf3f1f43-f34e-4fec-b22e-7abe0484d800\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk"
Jul 06 21:18:18.219169 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:18.219071    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf3f1f43-f34e-4fec-b22e-7abe0484d800-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk\" (UID: \"bf3f1f43-f34e-4fec-b22e-7abe0484d800\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk"
Jul 06 21:18:18.219169 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:18.219098    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1f43-f34e-4fec-b22e-7abe0484d800-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk\" (UID: \"bf3f1f43-f34e-4fec-b22e-7abe0484d800\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk"
Jul 06 21:18:18.320000 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:18.319969    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf3f1f43-f34e-4fec-b22e-7abe0484d800-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk\" (UID: \"bf3f1f43-f34e-4fec-b22e-7abe0484d800\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk"
Jul 06 21:18:18.320122 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:18.320005    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1f43-f34e-4fec-b22e-7abe0484d800-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk\" (UID: \"bf3f1f43-f34e-4fec-b22e-7abe0484d800\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk"
Jul 06 21:18:18.320122 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:18.320049    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpkvm\" (UniqueName: \"kubernetes.io/projected/bf3f1f43-f34e-4fec-b22e-7abe0484d800-kube-api-access-vpkvm\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk\" (UID: \"bf3f1f43-f34e-4fec-b22e-7abe0484d800\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk"
Jul 06 21:18:18.320440 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:18.320421    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1f43-f34e-4fec-b22e-7abe0484d800-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk\" (UID: \"bf3f1f43-f34e-4fec-b22e-7abe0484d800\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk"
Jul 06 21:18:18.322463 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:18.322446    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf3f1f43-f34e-4fec-b22e-7abe0484d800-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk\" (UID: \"bf3f1f43-f34e-4fec-b22e-7abe0484d800\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk"
Jul 06 21:18:18.330080 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:18.330060    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpkvm\" (UniqueName: \"kubernetes.io/projected/bf3f1f43-f34e-4fec-b22e-7abe0484d800-kube-api-access-vpkvm\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk\" (UID: \"bf3f1f43-f34e-4fec-b22e-7abe0484d800\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk"
Jul 06 21:18:18.443827 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:18.443771    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk"
Jul 06 21:18:18.571712 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:18.571688    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk"]
Jul 06 21:18:18.574166 ip-10-0-134-202 kubenswrapper[2559]: W0706 21:18:18.574140    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3f1f43_f34e_4fec_b22e_7abe0484d800.slice/crio-4fd016bdf712cb39c42e3dfcc211c768fd0c4e3fddcaa5f4e534e85e19dd6e63 WatchSource:0}: Error finding container 4fd016bdf712cb39c42e3dfcc211c768fd0c4e3fddcaa5f4e534e85e19dd6e63: Status 404 returned error can't find the container with id 4fd016bdf712cb39c42e3dfcc211c768fd0c4e3fddcaa5f4e534e85e19dd6e63
Jul 06 21:18:19.563648 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:19.563596    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk" event={"ID":"bf3f1f43-f34e-4fec-b22e-7abe0484d800","Type":"ContainerStarted","Data":"2424c45ee071ee1dd56da8fb91c14812abc839a6e000d4a6eea247a5623d6e9d"}
Jul 06 21:18:19.564047 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:19.563669    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk" event={"ID":"bf3f1f43-f34e-4fec-b22e-7abe0484d800","Type":"ContainerStarted","Data":"4fd016bdf712cb39c42e3dfcc211c768fd0c4e3fddcaa5f4e534e85e19dd6e63"}
Jul 06 21:18:19.564047 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:19.563757    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk"
Jul 06 21:18:19.585472 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:19.585432    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk" podStartSLOduration=1.585417007 podStartE2EDuration="1.585417007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:18:19.584748241 +0000 UTC m=+1748.419635627" watchObservedRunningTime="2026-07-06 21:18:19.585417007 +0000 UTC m=+1748.420304363"
Jul 06 21:18:50.570846 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:18:50.570820    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk"
Jul 06 21:20:00.026697 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.026661    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"]
Jul 06 21:20:00.030039 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.030023    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.032236 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.032203    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 06 21:20:00.032356 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.032288    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-vdr6g\""
Jul 06 21:20:00.041355 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.041334    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"]
Jul 06 21:20:00.154829 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.154798    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.154998 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.154843    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.154998 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.154883    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.154998 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.154949    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.154998 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.154982    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.155138 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.155013    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq2f2\" (UniqueName: \"kubernetes.io/projected/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-kube-api-access-gq2f2\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.155138 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.155060    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.155138 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.155083    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.155138 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.155102    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.256215 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.256187    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.256215 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.256221    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.256431 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.256244    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq2f2\" (UniqueName: \"kubernetes.io/projected/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-kube-api-access-gq2f2\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.256431 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.256283    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.256431 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.256314    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.256431 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.256353    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.256681 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.256470    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.256681 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.256533    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.256681 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.256591    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.256843 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.256697    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.256843 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.256728    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.256994 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.256968    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.257116 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.257008    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.257116 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.257043    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.258662 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.258642    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.259019 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.258998    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.264570 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.264551    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.267136 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.267113    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq2f2\" (UniqueName: \"kubernetes.io/projected/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-kube-api-access-gq2f2\") pod \"router-gateway-1-openshift-default-799f46c59b-g9fx9\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.340416 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.340390    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:00.479437 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.479405    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"]
Jul 06 21:20:00.480412 ip-10-0-134-202 kubenswrapper[2559]: W0706 21:20:00.480386    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3a2172_67b0_4c37_aee4_6f6eae0bd831.slice/crio-5ffd1d56b593ec6a896a30298f1c1564347856efa469def0ba0668b3d2a8459c WatchSource:0}: Error finding container 5ffd1d56b593ec6a896a30298f1c1564347856efa469def0ba0668b3d2a8459c: Status 404 returned error can't find the container with id 5ffd1d56b593ec6a896a30298f1c1564347856efa469def0ba0668b3d2a8459c
Jul 06 21:20:00.482460 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.482429    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:20:00.482546 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.482492    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:20:00.482546 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:00.482520    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:20:01.013526 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:01.013491    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9" event={"ID":"ea3a2172-67b0-4c37-aee4-6f6eae0bd831","Type":"ContainerStarted","Data":"bada2f227ec3b1a5e64d6a8084f1bf460940df67970536432c248c7524834c71"}
Jul 06 21:20:01.013717 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:01.013553    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9" event={"ID":"ea3a2172-67b0-4c37-aee4-6f6eae0bd831","Type":"ContainerStarted","Data":"5ffd1d56b593ec6a896a30298f1c1564347856efa469def0ba0668b3d2a8459c"}
Jul 06 21:20:01.040352 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:01.040301    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9" podStartSLOduration=1.040284126 podStartE2EDuration="1.040284126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:20:01.038138348 +0000 UTC m=+1849.873025735" watchObservedRunningTime="2026-07-06 21:20:01.040284126 +0000 UTC m=+1849.875171496"
Jul 06 21:20:01.341396 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:01.341364    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:01.346094 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:01.346074    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:02.017847 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.017815    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:02.018965 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.018945    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:02.234352 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.234316    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"]
Jul 06 21:20:02.237925 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.237910    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.249747 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.249727    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"]
Jul 06 21:20:02.276048 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.275980    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76097b92-270c-426b-b7ec-95a0dfed13e2-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.276048 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.276012    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76097b92-270c-426b-b7ec-95a0dfed13e2-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.276217 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.276051    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76097b92-270c-426b-b7ec-95a0dfed13e2-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.276217 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.276098    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76097b92-270c-426b-b7ec-95a0dfed13e2-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.276217 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.276187    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76097b92-270c-426b-b7ec-95a0dfed13e2-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.276372 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.276223    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76097b92-270c-426b-b7ec-95a0dfed13e2-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.276372 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.276244    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76097b92-270c-426b-b7ec-95a0dfed13e2-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.276372 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.276262    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7fhx\" (UniqueName: \"kubernetes.io/projected/76097b92-270c-426b-b7ec-95a0dfed13e2-kube-api-access-q7fhx\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.276372 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.276330    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76097b92-270c-426b-b7ec-95a0dfed13e2-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.377459 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.377428    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76097b92-270c-426b-b7ec-95a0dfed13e2-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.377589 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.377468    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7fhx\" (UniqueName: \"kubernetes.io/projected/76097b92-270c-426b-b7ec-95a0dfed13e2-kube-api-access-q7fhx\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.377589 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.377511    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76097b92-270c-426b-b7ec-95a0dfed13e2-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.377589 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.377562    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76097b92-270c-426b-b7ec-95a0dfed13e2-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.377800 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.377592    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76097b92-270c-426b-b7ec-95a0dfed13e2-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.377800 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.377695    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76097b92-270c-426b-b7ec-95a0dfed13e2-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.377800 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.377722    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76097b92-270c-426b-b7ec-95a0dfed13e2-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.377800 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.377750    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76097b92-270c-426b-b7ec-95a0dfed13e2-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.377800 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.377789    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76097b92-270c-426b-b7ec-95a0dfed13e2-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.378061 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.377949    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76097b92-270c-426b-b7ec-95a0dfed13e2-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.378061 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.378000    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76097b92-270c-426b-b7ec-95a0dfed13e2-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.378166 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.378146    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76097b92-270c-426b-b7ec-95a0dfed13e2-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.378217 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.378188    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76097b92-270c-426b-b7ec-95a0dfed13e2-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.378495 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.378472    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76097b92-270c-426b-b7ec-95a0dfed13e2-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.379911 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.379884    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76097b92-270c-426b-b7ec-95a0dfed13e2-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.380292 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.380272    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76097b92-270c-426b-b7ec-95a0dfed13e2-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.390344 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.390317    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7fhx\" (UniqueName: \"kubernetes.io/projected/76097b92-270c-426b-b7ec-95a0dfed13e2-kube-api-access-q7fhx\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.391438 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.391421    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76097b92-270c-426b-b7ec-95a0dfed13e2-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-zpwwt\" (UID: \"76097b92-270c-426b-b7ec-95a0dfed13e2\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.549684 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.549658    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:02.685776 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.685746    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"]
Jul 06 21:20:02.688441 ip-10-0-134-202 kubenswrapper[2559]: W0706 21:20:02.688417    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76097b92_270c_426b_b7ec_95a0dfed13e2.slice/crio-98409c1e6cf9459eb060b882f50977eba53ffcd36eee02f3226d58637d705452 WatchSource:0}: Error finding container 98409c1e6cf9459eb060b882f50977eba53ffcd36eee02f3226d58637d705452: Status 404 returned error can't find the container with id 98409c1e6cf9459eb060b882f50977eba53ffcd36eee02f3226d58637d705452
Jul 06 21:20:02.690546 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.690507    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:20:02.690666 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.690597    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:20:02.690737 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:02.690701    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:20:03.023320 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:03.023231    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt" event={"ID":"76097b92-270c-426b-b7ec-95a0dfed13e2","Type":"ContainerStarted","Data":"c50cf7641af59d4dd6720f32b840a43ac9f2e6044398e4eab267e0a686cdab7b"}
Jul 06 21:20:03.023320 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:03.023286    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt" event={"ID":"76097b92-270c-426b-b7ec-95a0dfed13e2","Type":"ContainerStarted","Data":"98409c1e6cf9459eb060b882f50977eba53ffcd36eee02f3226d58637d705452"}
Jul 06 21:20:03.047485 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:03.047427    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt" podStartSLOduration=1.047409287 podStartE2EDuration="1.047409287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:20:03.046424782 +0000 UTC m=+1851.881312172" watchObservedRunningTime="2026-07-06 21:20:03.047409287 +0000 UTC m=+1851.882296658"
Jul 06 21:20:03.550694 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:03.550651    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:03.552190 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:03.552163    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt" podUID="76097b92-270c-426b-b7ec-95a0dfed13e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 06 21:20:04.550820 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:04.550776    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt" podUID="76097b92-270c-426b-b7ec-95a0dfed13e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 06 21:20:05.550376 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:05.550320    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt" podUID="76097b92-270c-426b-b7ec-95a0dfed13e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 06 21:20:06.550580 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:06.550538    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt" podUID="76097b92-270c-426b-b7ec-95a0dfed13e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 06 21:20:07.550846 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:07.550801    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt" podUID="76097b92-270c-426b-b7ec-95a0dfed13e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 06 21:20:08.550408 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:08.550358    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt" podUID="76097b92-270c-426b-b7ec-95a0dfed13e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 06 21:20:09.555400 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:09.555371    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:09.555888 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:09.555753    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:09.556635 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:09.556603    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt"
Jul 06 21:20:09.651261 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:09.651221    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"]
Jul 06 21:20:09.651526 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:09.651493    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9" podUID="ea3a2172-67b0-4c37-aee4-6f6eae0bd831" containerName="istio-proxy" containerID="cri-o://bada2f227ec3b1a5e64d6a8084f1bf460940df67970536432c248c7524834c71" gracePeriod=30
Jul 06 21:20:12.900695 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:12.900668    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:12.985409 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:12.985328    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-token\") pod \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") "
Jul 06 21:20:12.985409 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:12.985368    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-workload-socket\") pod \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") "
Jul 06 21:20:12.985409 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:12.985397    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istiod-ca-cert\") pod \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") "
Jul 06 21:20:12.985690 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:12.985421    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gq2f2\" (UniqueName: \"kubernetes.io/projected/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-kube-api-access-gq2f2\") pod \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") "
Jul 06 21:20:12.985690 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:12.985477    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-credential-socket\") pod \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") "
Jul 06 21:20:12.985690 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:12.985508    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-podinfo\") pod \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") "
Jul 06 21:20:12.985690 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:12.985539    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-workload-certs\") pod \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") "
Jul 06 21:20:12.985690 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:12.985583    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-envoy\") pod \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") "
Jul 06 21:20:12.985690 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:12.985631    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-data\") pod \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\" (UID: \"ea3a2172-67b0-4c37-aee4-6f6eae0bd831\") "
Jul 06 21:20:12.985690 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:12.985682    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ea3a2172-67b0-4c37-aee4-6f6eae0bd831" (UID: "ea3a2172-67b0-4c37-aee4-6f6eae0bd831"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:20:12.986064 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:12.985828    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ea3a2172-67b0-4c37-aee4-6f6eae0bd831" (UID: "ea3a2172-67b0-4c37-aee4-6f6eae0bd831"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:20:12.986064 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:12.985858    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ea3a2172-67b0-4c37-aee4-6f6eae0bd831" (UID: "ea3a2172-67b0-4c37-aee4-6f6eae0bd831"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:20:12.986064 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:12.985950    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-credential-socket\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:20:12.986064 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:12.986033    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-data" (OuterVolumeSpecName: "istio-data") pod "ea3a2172-67b0-4c37-aee4-6f6eae0bd831" (UID: "ea3a2172-67b0-4c37-aee4-6f6eae0bd831"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:20:12.986274 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:12.986135    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ea3a2172-67b0-4c37-aee4-6f6eae0bd831" (UID: "ea3a2172-67b0-4c37-aee4-6f6eae0bd831"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:20:12.986274 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:12.986164    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-workload-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:20:12.986274 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:12.986180    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-workload-socket\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:20:12.987724 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:12.987697    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-token" (OuterVolumeSpecName: "istio-token") pod "ea3a2172-67b0-4c37-aee4-6f6eae0bd831" (UID: "ea3a2172-67b0-4c37-aee4-6f6eae0bd831"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:20:12.987829 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:12.987734    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ea3a2172-67b0-4c37-aee4-6f6eae0bd831" (UID: "ea3a2172-67b0-4c37-aee4-6f6eae0bd831"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 21:20:12.987829 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:12.987735    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-kube-api-access-gq2f2" (OuterVolumeSpecName: "kube-api-access-gq2f2") pod "ea3a2172-67b0-4c37-aee4-6f6eae0bd831" (UID: "ea3a2172-67b0-4c37-aee4-6f6eae0bd831"). InnerVolumeSpecName "kube-api-access-gq2f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:20:12.987829 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:12.987805    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ea3a2172-67b0-4c37-aee4-6f6eae0bd831" (UID: "ea3a2172-67b0-4c37-aee4-6f6eae0bd831"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:20:13.084987 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:13.084957    2559 generic.go:356] "Generic (PLEG): container finished" podID="ea3a2172-67b0-4c37-aee4-6f6eae0bd831" containerID="bada2f227ec3b1a5e64d6a8084f1bf460940df67970536432c248c7524834c71" exitCode=0
Jul 06 21:20:13.085156 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:13.085001    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"
Jul 06 21:20:13.085156 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:13.085041    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9" event={"ID":"ea3a2172-67b0-4c37-aee4-6f6eae0bd831","Type":"ContainerDied","Data":"bada2f227ec3b1a5e64d6a8084f1bf460940df67970536432c248c7524834c71"}
Jul 06 21:20:13.085156 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:13.085099    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9" event={"ID":"ea3a2172-67b0-4c37-aee4-6f6eae0bd831","Type":"ContainerDied","Data":"5ffd1d56b593ec6a896a30298f1c1564347856efa469def0ba0668b3d2a8459c"}
Jul 06 21:20:13.085156 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:13.085125    2559 scope.go:117] "RemoveContainer" containerID="bada2f227ec3b1a5e64d6a8084f1bf460940df67970536432c248c7524834c71"
Jul 06 21:20:13.086671 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:13.086655    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-token\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:20:13.086795 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:13.086675    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istiod-ca-cert\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:20:13.086795 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:13.086690    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq2f2\" (UniqueName: \"kubernetes.io/projected/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-kube-api-access-gq2f2\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:20:13.086795 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:13.086699    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-podinfo\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:20:13.086795 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:13.086709    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-envoy\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:20:13.086795 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:13.086718    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3a2172-67b0-4c37-aee4-6f6eae0bd831-istio-data\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:20:13.094414 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:13.094397    2559 scope.go:117] "RemoveContainer" containerID="bada2f227ec3b1a5e64d6a8084f1bf460940df67970536432c248c7524834c71"
Jul 06 21:20:13.094681 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:20:13.094654    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bada2f227ec3b1a5e64d6a8084f1bf460940df67970536432c248c7524834c71\": container with ID starting with bada2f227ec3b1a5e64d6a8084f1bf460940df67970536432c248c7524834c71 not found: ID does not exist" containerID="bada2f227ec3b1a5e64d6a8084f1bf460940df67970536432c248c7524834c71"
Jul 06 21:20:13.094758 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:13.094691    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bada2f227ec3b1a5e64d6a8084f1bf460940df67970536432c248c7524834c71"} err="failed to get container status \"bada2f227ec3b1a5e64d6a8084f1bf460940df67970536432c248c7524834c71\": rpc error: code = NotFound desc = could not find container \"bada2f227ec3b1a5e64d6a8084f1bf460940df67970536432c248c7524834c71\": container with ID starting with bada2f227ec3b1a5e64d6a8084f1bf460940df67970536432c248c7524834c71 not found: ID does not exist"
Jul 06 21:20:13.115119 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:13.115098    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"]
Jul 06 21:20:13.123145 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:13.123125    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9"]
Jul 06 21:20:13.657846 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:20:13.657800    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea3a2172-67b0-4c37-aee4-6f6eae0bd831" path="/var/lib/kubelet/pods/ea3a2172-67b0-4c37-aee4-6f6eae0bd831/volumes"
Jul 06 21:21:22.421892 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:22.421861    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk"]
Jul 06 21:21:22.430522 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:21:22.430490    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:21:22.430522 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:21:22.430522 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 21:21:22.430522 ip-10-0-134-202 kubenswrapper[2559]:  > containerID="2424c45ee071ee1dd56da8fb91c14812abc839a6e000d4a6eea247a5623d6e9d" cmd=["/bin/sleep","15"]
Jul 06 21:21:22.430799 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:22.430537    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk" message="" err=<
Jul 06 21:21:22.430799 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:21:22.430799 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 21:21:22.430799 ip-10-0-134-202 kubenswrapper[2559]:  >
Jul 06 21:21:22.430799 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:21:22.430552    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:21:22.430799 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:21:22.430799 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 21:21:22.430799 ip-10-0-134-202 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk" podUID="bf3f1f43-f34e-4fec-b22e-7abe0484d800" containerName="main" containerID="cri-o://2424c45ee071ee1dd56da8fb91c14812abc839a6e000d4a6eea247a5623d6e9d"
Jul 06 21:21:22.430799 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:22.430596    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk" podUID="bf3f1f43-f34e-4fec-b22e-7abe0484d800" containerName="main" containerID="cri-o://2424c45ee071ee1dd56da8fb91c14812abc839a6e000d4a6eea247a5623d6e9d" gracePeriod=60
Jul 06 21:21:22.675388 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:22.675329    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk"
Jul 06 21:21:22.781456 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:22.781431    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpkvm\" (UniqueName: \"kubernetes.io/projected/bf3f1f43-f34e-4fec-b22e-7abe0484d800-kube-api-access-vpkvm\") pod \"bf3f1f43-f34e-4fec-b22e-7abe0484d800\" (UID: \"bf3f1f43-f34e-4fec-b22e-7abe0484d800\") "
Jul 06 21:21:22.781631 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:22.781481    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1f43-f34e-4fec-b22e-7abe0484d800-tokenizer-uds\") pod \"bf3f1f43-f34e-4fec-b22e-7abe0484d800\" (UID: \"bf3f1f43-f34e-4fec-b22e-7abe0484d800\") "
Jul 06 21:21:22.781631 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:22.781516    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf3f1f43-f34e-4fec-b22e-7abe0484d800-tls-certs\") pod \"bf3f1f43-f34e-4fec-b22e-7abe0484d800\" (UID: \"bf3f1f43-f34e-4fec-b22e-7abe0484d800\") "
Jul 06 21:21:22.781815 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:22.781793    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf3f1f43-f34e-4fec-b22e-7abe0484d800-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "bf3f1f43-f34e-4fec-b22e-7abe0484d800" (UID: "bf3f1f43-f34e-4fec-b22e-7abe0484d800"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:21:22.783517 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:22.783485    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf3f1f43-f34e-4fec-b22e-7abe0484d800-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf3f1f43-f34e-4fec-b22e-7abe0484d800" (UID: "bf3f1f43-f34e-4fec-b22e-7abe0484d800"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:21:22.783604 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:22.783546    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf3f1f43-f34e-4fec-b22e-7abe0484d800-kube-api-access-vpkvm" (OuterVolumeSpecName: "kube-api-access-vpkvm") pod "bf3f1f43-f34e-4fec-b22e-7abe0484d800" (UID: "bf3f1f43-f34e-4fec-b22e-7abe0484d800"). InnerVolumeSpecName "kube-api-access-vpkvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:21:22.882664 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:22.882606    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpkvm\" (UniqueName: \"kubernetes.io/projected/bf3f1f43-f34e-4fec-b22e-7abe0484d800-kube-api-access-vpkvm\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:21:22.882664 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:22.882660    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1f43-f34e-4fec-b22e-7abe0484d800-tokenizer-uds\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:21:22.882664 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:22.882671    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf3f1f43-f34e-4fec-b22e-7abe0484d800-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:21:23.410002 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:23.409961    2559 generic.go:356] "Generic (PLEG): container finished" podID="bf3f1f43-f34e-4fec-b22e-7abe0484d800" containerID="2424c45ee071ee1dd56da8fb91c14812abc839a6e000d4a6eea247a5623d6e9d" exitCode=0
Jul 06 21:21:23.410179 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:23.410012    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk"
Jul 06 21:21:23.410179 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:23.410013    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk" event={"ID":"bf3f1f43-f34e-4fec-b22e-7abe0484d800","Type":"ContainerDied","Data":"2424c45ee071ee1dd56da8fb91c14812abc839a6e000d4a6eea247a5623d6e9d"}
Jul 06 21:21:23.410179 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:23.410062    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk" event={"ID":"bf3f1f43-f34e-4fec-b22e-7abe0484d800","Type":"ContainerDied","Data":"4fd016bdf712cb39c42e3dfcc211c768fd0c4e3fddcaa5f4e534e85e19dd6e63"}
Jul 06 21:21:23.410179 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:23.410092    2559 scope.go:117] "RemoveContainer" containerID="2424c45ee071ee1dd56da8fb91c14812abc839a6e000d4a6eea247a5623d6e9d"
Jul 06 21:21:23.419284 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:23.419265    2559 scope.go:117] "RemoveContainer" containerID="2424c45ee071ee1dd56da8fb91c14812abc839a6e000d4a6eea247a5623d6e9d"
Jul 06 21:21:23.419536 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:21:23.419514    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2424c45ee071ee1dd56da8fb91c14812abc839a6e000d4a6eea247a5623d6e9d\": container with ID starting with 2424c45ee071ee1dd56da8fb91c14812abc839a6e000d4a6eea247a5623d6e9d not found: ID does not exist" containerID="2424c45ee071ee1dd56da8fb91c14812abc839a6e000d4a6eea247a5623d6e9d"
Jul 06 21:21:23.419576 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:23.419546    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2424c45ee071ee1dd56da8fb91c14812abc839a6e000d4a6eea247a5623d6e9d"} err="failed to get container status \"2424c45ee071ee1dd56da8fb91c14812abc839a6e000d4a6eea247a5623d6e9d\": rpc error: code = NotFound desc = could not find container \"2424c45ee071ee1dd56da8fb91c14812abc839a6e000d4a6eea247a5623d6e9d\": container with ID starting with 2424c45ee071ee1dd56da8fb91c14812abc839a6e000d4a6eea247a5623d6e9d not found: ID does not exist"
Jul 06 21:21:23.431307 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:23.431280    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk"]
Jul 06 21:21:23.437351 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:23.437325    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk"]
Jul 06 21:21:23.657960 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:23.657930    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf3f1f43-f34e-4fec-b22e-7abe0484d800" path="/var/lib/kubelet/pods/bf3f1f43-f34e-4fec-b22e-7abe0484d800/volumes"
Jul 06 21:21:50.846339 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:50.846301    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv"]
Jul 06 21:21:50.846826 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:50.846658    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf3f1f43-f34e-4fec-b22e-7abe0484d800" containerName="main"
Jul 06 21:21:50.846826 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:50.846670    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf3f1f43-f34e-4fec-b22e-7abe0484d800" containerName="main"
Jul 06 21:21:50.846826 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:50.846693    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea3a2172-67b0-4c37-aee4-6f6eae0bd831" containerName="istio-proxy"
Jul 06 21:21:50.846826 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:50.846698    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea3a2172-67b0-4c37-aee4-6f6eae0bd831" containerName="istio-proxy"
Jul 06 21:21:50.846826 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:50.846759    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf3f1f43-f34e-4fec-b22e-7abe0484d800" containerName="main"
Jul 06 21:21:50.846826 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:50.846765    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea3a2172-67b0-4c37-aee4-6f6eae0bd831" containerName="istio-proxy"
Jul 06 21:21:50.849733 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:50.849712    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv"
Jul 06 21:21:50.852769 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:50.852746    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 06 21:21:50.854641 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:50.854594    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-6tmzx\""
Jul 06 21:21:50.855282 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:50.855262    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g6qfk\""
Jul 06 21:21:50.864991 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:50.864973    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv"]
Jul 06 21:21:50.907136 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:50.907113    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/029c2f6a-b869-4b84-b178-3b7dc3eef6fe-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv\" (UID: \"029c2f6a-b869-4b84-b178-3b7dc3eef6fe\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv"
Jul 06 21:21:50.907242 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:50.907181    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxbhn\" (UniqueName: \"kubernetes.io/projected/029c2f6a-b869-4b84-b178-3b7dc3eef6fe-kube-api-access-pxbhn\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv\" (UID: \"029c2f6a-b869-4b84-b178-3b7dc3eef6fe\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv"
Jul 06 21:21:50.907294 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:50.907248    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/029c2f6a-b869-4b84-b178-3b7dc3eef6fe-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv\" (UID: \"029c2f6a-b869-4b84-b178-3b7dc3eef6fe\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv"
Jul 06 21:21:51.008278 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:51.008239    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxbhn\" (UniqueName: \"kubernetes.io/projected/029c2f6a-b869-4b84-b178-3b7dc3eef6fe-kube-api-access-pxbhn\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv\" (UID: \"029c2f6a-b869-4b84-b178-3b7dc3eef6fe\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv"
Jul 06 21:21:51.008470 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:51.008305    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/029c2f6a-b869-4b84-b178-3b7dc3eef6fe-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv\" (UID: \"029c2f6a-b869-4b84-b178-3b7dc3eef6fe\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv"
Jul 06 21:21:51.008470 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:51.008344    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/029c2f6a-b869-4b84-b178-3b7dc3eef6fe-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv\" (UID: \"029c2f6a-b869-4b84-b178-3b7dc3eef6fe\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv"
Jul 06 21:21:51.008785 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:51.008762    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/029c2f6a-b869-4b84-b178-3b7dc3eef6fe-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv\" (UID: \"029c2f6a-b869-4b84-b178-3b7dc3eef6fe\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv"
Jul 06 21:21:51.010685 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:51.010665    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/029c2f6a-b869-4b84-b178-3b7dc3eef6fe-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv\" (UID: \"029c2f6a-b869-4b84-b178-3b7dc3eef6fe\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv"
Jul 06 21:21:51.018088 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:51.018066    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxbhn\" (UniqueName: \"kubernetes.io/projected/029c2f6a-b869-4b84-b178-3b7dc3eef6fe-kube-api-access-pxbhn\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv\" (UID: \"029c2f6a-b869-4b84-b178-3b7dc3eef6fe\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv"
Jul 06 21:21:51.158992 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:51.158908    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv"
Jul 06 21:21:51.289854 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:51.289823    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv"]
Jul 06 21:21:51.291078 ip-10-0-134-202 kubenswrapper[2559]: W0706 21:21:51.291048    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029c2f6a_b869_4b84_b178_3b7dc3eef6fe.slice/crio-563b63bce10c608170a40cbd6117b4d1478d7ca78b9a12b4ab34225021c0a55a WatchSource:0}: Error finding container 563b63bce10c608170a40cbd6117b4d1478d7ca78b9a12b4ab34225021c0a55a: Status 404 returned error can't find the container with id 563b63bce10c608170a40cbd6117b4d1478d7ca78b9a12b4ab34225021c0a55a
Jul 06 21:21:51.537120 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:51.537087    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv" event={"ID":"029c2f6a-b869-4b84-b178-3b7dc3eef6fe","Type":"ContainerStarted","Data":"9c4b2ac68a4b7ee75f6286939ac804d00da4f8f24ee360093df5172e0013699c"}
Jul 06 21:21:51.537283 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:51.537135    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv" event={"ID":"029c2f6a-b869-4b84-b178-3b7dc3eef6fe","Type":"ContainerStarted","Data":"563b63bce10c608170a40cbd6117b4d1478d7ca78b9a12b4ab34225021c0a55a"}
Jul 06 21:21:51.537283 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:51.537215    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv"
Jul 06 21:21:51.564517 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:21:51.564463    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv" podStartSLOduration=1.5644436179999999 podStartE2EDuration="1.564443618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:21:51.562994621 +0000 UTC m=+1960.397882008" watchObservedRunningTime="2026-07-06 21:21:51.564443618 +0000 UTC m=+1960.399330986"
Jul 06 21:22:22.543045 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:22:22.543018    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv"
Jul 06 21:31:00.418010 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.417970    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"]
Jul 06 21:31:00.421754 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.421733    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.423977 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.423949    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-2-openshift-default-dockercfg-md8ln\""
Jul 06 21:31:00.434423 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.434402    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"]
Jul 06 21:31:00.566225 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.566188    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-credential-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.566225 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.566227    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-podinfo\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.566491 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.566249    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.566491 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.566303    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-workload-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.566491 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.566402    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-token\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.566491 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.566448    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xn7v\" (UniqueName: \"kubernetes.io/projected/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-kube-api-access-8xn7v\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.566716 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.566498    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-data\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.566716 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.566514    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-workload-certs\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.566716 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.566529    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-envoy\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.667854 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.667809    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-token\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.667854 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.667856    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xn7v\" (UniqueName: \"kubernetes.io/projected/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-kube-api-access-8xn7v\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.668157 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.667896    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-data\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.668157 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.667915    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-workload-certs\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.668157 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.667936    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-envoy\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.668157 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.668073    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-credential-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.668157 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.668118    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-podinfo\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.668157 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.668142    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.668467 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.668180    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-workload-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.668467 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.668295    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-data\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.668467 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.668352    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-workload-certs\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.668467 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.668418    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-credential-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.668707 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.668532    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-workload-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.668961 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.668939    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.670152 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.670131    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-envoy\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.670552 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.670531    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-podinfo\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.675997 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.675974    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-token\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.676744 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.676724    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xn7v\" (UniqueName: \"kubernetes.io/projected/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-kube-api-access-8xn7v\") pod \"router-gateway-2-openshift-default-54c789bdc6-fj8fg\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.732733 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.732698    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:00.871450 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.871420    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"]
Jul 06 21:31:00.873283 ip-10-0-134-202 kubenswrapper[2559]: W0706 21:31:00.873252    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1b720f_20bc_4c1b_b075_d7b3df0d5253.slice/crio-32b12de950b5f291ff91cac1fe86c3506eecb2cb6ceb7c435422bc6bd1fdd564 WatchSource:0}: Error finding container 32b12de950b5f291ff91cac1fe86c3506eecb2cb6ceb7c435422bc6bd1fdd564: Status 404 returned error can't find the container with id 32b12de950b5f291ff91cac1fe86c3506eecb2cb6ceb7c435422bc6bd1fdd564
Jul 06 21:31:00.875698 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.875665    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:31:00.875781 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.875730    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:31:00.875781 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:00.875759    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:31:01.072684 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:01.072646    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg" event={"ID":"bc1b720f-20bc-4c1b-b075-d7b3df0d5253","Type":"ContainerStarted","Data":"7fe9c47a653ff235260d058e7b06374d79513caba8f43e9b490d54c65bbc6c2c"}
Jul 06 21:31:01.072848 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:01.072707    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg" event={"ID":"bc1b720f-20bc-4c1b-b075-d7b3df0d5253","Type":"ContainerStarted","Data":"32b12de950b5f291ff91cac1fe86c3506eecb2cb6ceb7c435422bc6bd1fdd564"}
Jul 06 21:31:01.095120 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:01.095055    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg" podStartSLOduration=1.095032752 podStartE2EDuration="1.095032752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:31:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:31:01.091843332 +0000 UTC m=+2509.926730697" watchObservedRunningTime="2026-07-06 21:31:01.095032752 +0000 UTC m=+2509.929920119"
Jul 06 21:31:01.733873 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:01.733835    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:01.738561 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:01.738537    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:02.078598 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:02.078549    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:02.079521 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:02.079501    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:03.860946 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:03.860868    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"]
Jul 06 21:31:03.864296 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:03.864279    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:03.877762 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:03.877737    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"]
Jul 06 21:31:03.995659 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:03.995601    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/047c1603-6763-4cb1-8c0f-8094f55a94b7-istio-envoy\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:03.995659 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:03.995659    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/047c1603-6763-4cb1-8c0f-8094f55a94b7-istio-token\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:03.995881 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:03.995712    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l4xb\" (UniqueName: \"kubernetes.io/projected/047c1603-6763-4cb1-8c0f-8094f55a94b7-kube-api-access-8l4xb\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:03.995881 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:03.995745    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/047c1603-6763-4cb1-8c0f-8094f55a94b7-workload-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:03.995881 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:03.995771    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/047c1603-6763-4cb1-8c0f-8094f55a94b7-istio-data\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:03.995881 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:03.995808    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/047c1603-6763-4cb1-8c0f-8094f55a94b7-istio-podinfo\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:03.995881 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:03.995845    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/047c1603-6763-4cb1-8c0f-8094f55a94b7-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:03.996037 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:03.995892    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/047c1603-6763-4cb1-8c0f-8094f55a94b7-credential-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:03.996037 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:03.995927    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/047c1603-6763-4cb1-8c0f-8094f55a94b7-workload-certs\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:04.096574 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.096541    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/047c1603-6763-4cb1-8c0f-8094f55a94b7-istio-envoy\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:04.096574 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.096573    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/047c1603-6763-4cb1-8c0f-8094f55a94b7-istio-token\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:04.096816 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.096632    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l4xb\" (UniqueName: \"kubernetes.io/projected/047c1603-6763-4cb1-8c0f-8094f55a94b7-kube-api-access-8l4xb\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:04.096816 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.096659    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/047c1603-6763-4cb1-8c0f-8094f55a94b7-workload-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:04.096816 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.096676    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/047c1603-6763-4cb1-8c0f-8094f55a94b7-istio-data\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:04.096816 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.096719    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/047c1603-6763-4cb1-8c0f-8094f55a94b7-istio-podinfo\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:04.096816 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.096748    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/047c1603-6763-4cb1-8c0f-8094f55a94b7-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:04.096816 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.096779    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/047c1603-6763-4cb1-8c0f-8094f55a94b7-credential-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:04.097135 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.096820    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/047c1603-6763-4cb1-8c0f-8094f55a94b7-workload-certs\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:04.097192 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.097143    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/047c1603-6763-4cb1-8c0f-8094f55a94b7-istio-data\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:04.097192 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.097172    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/047c1603-6763-4cb1-8c0f-8094f55a94b7-workload-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:04.097297 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.097277    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/047c1603-6763-4cb1-8c0f-8094f55a94b7-workload-certs\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:04.097401 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.097382    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/047c1603-6763-4cb1-8c0f-8094f55a94b7-credential-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:04.097559 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.097533    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/047c1603-6763-4cb1-8c0f-8094f55a94b7-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:04.099091 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.099074    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/047c1603-6763-4cb1-8c0f-8094f55a94b7-istio-envoy\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:04.099221 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.099203    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/047c1603-6763-4cb1-8c0f-8094f55a94b7-istio-podinfo\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:04.105735 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.105709    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l4xb\" (UniqueName: \"kubernetes.io/projected/047c1603-6763-4cb1-8c0f-8094f55a94b7-kube-api-access-8l4xb\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:04.105843 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.105826    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/047c1603-6763-4cb1-8c0f-8094f55a94b7-istio-token\") pod \"router-gateway-2-openshift-default-684dc9b4b9-nrf44\" (UID: \"047c1603-6763-4cb1-8c0f-8094f55a94b7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:04.174757 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.174677    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:04.318270 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.318244    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"]
Jul 06 21:31:04.320343 ip-10-0-134-202 kubenswrapper[2559]: W0706 21:31:04.320317    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047c1603_6763_4cb1_8c0f_8094f55a94b7.slice/crio-a7f5669caf8b826630ed61cb6cac262cb16ba23f8edfb42b6932011a05ad46bf WatchSource:0}: Error finding container a7f5669caf8b826630ed61cb6cac262cb16ba23f8edfb42b6932011a05ad46bf: Status 404 returned error can't find the container with id a7f5669caf8b826630ed61cb6cac262cb16ba23f8edfb42b6932011a05ad46bf
Jul 06 21:31:04.323049 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.323019    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:31:04.323141 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.323081    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:31:04.323141 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:04.323113    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:31:05.093684 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:05.093653    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44" event={"ID":"047c1603-6763-4cb1-8c0f-8094f55a94b7","Type":"ContainerStarted","Data":"cf7a5bd753fd3534a0f5324784e67821b78f5444d0487b8fc2e94e5c9f81a454"}
Jul 06 21:31:05.094056 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:05.093703    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44" event={"ID":"047c1603-6763-4cb1-8c0f-8094f55a94b7","Type":"ContainerStarted","Data":"a7f5669caf8b826630ed61cb6cac262cb16ba23f8edfb42b6932011a05ad46bf"}
Jul 06 21:31:05.175328 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:05.175290    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:05.179995 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:05.179972    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:05.203954 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:05.203908    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44" podStartSLOduration=2.203893858 podStartE2EDuration="2.203893858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:31:05.119874807 +0000 UTC m=+2513.954762176" watchObservedRunningTime="2026-07-06 21:31:05.203893858 +0000 UTC m=+2514.038781223"
Jul 06 21:31:06.098200 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:06.098162    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:06.099134 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:06.099116    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:06.153139 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:06.153104    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"]
Jul 06 21:31:06.153340 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:06.153316    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg" podUID="bc1b720f-20bc-4c1b-b075-d7b3df0d5253" containerName="istio-proxy" containerID="cri-o://7fe9c47a653ff235260d058e7b06374d79513caba8f43e9b490d54c65bbc6c2c" gracePeriod=30
Jul 06 21:31:14.706277 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.706252    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:14.719525 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.719501    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-credential-socket\") pod \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") "
Jul 06 21:31:14.719657 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.719539    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-token\") pod \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") "
Jul 06 21:31:14.719657 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.719580    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-data\") pod \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") "
Jul 06 21:31:14.719657 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.719602    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-workload-certs\") pod \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") "
Jul 06 21:31:14.719833 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.719657    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istiod-ca-cert\") pod \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") "
Jul 06 21:31:14.719833 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.719681    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-workload-socket\") pod \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") "
Jul 06 21:31:14.719833 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.719730    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-envoy\") pod \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") "
Jul 06 21:31:14.719833 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.719762    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-podinfo\") pod \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") "
Jul 06 21:31:14.719833 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.719786    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xn7v\" (UniqueName: \"kubernetes.io/projected/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-kube-api-access-8xn7v\") pod \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\" (UID: \"bc1b720f-20bc-4c1b-b075-d7b3df0d5253\") "
Jul 06 21:31:14.719833 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.719811    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bc1b720f-20bc-4c1b-b075-d7b3df0d5253" (UID: "bc1b720f-20bc-4c1b-b075-d7b3df0d5253"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:31:14.720141 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.719973    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bc1b720f-20bc-4c1b-b075-d7b3df0d5253" (UID: "bc1b720f-20bc-4c1b-b075-d7b3df0d5253"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:31:14.720195 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.720145    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-workload-socket\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:31:14.720195 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.720164    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-credential-socket\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:31:14.720415 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.720374    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bc1b720f-20bc-4c1b-b075-d7b3df0d5253" (UID: "bc1b720f-20bc-4c1b-b075-d7b3df0d5253"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:31:14.720501 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.720439    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-data" (OuterVolumeSpecName: "istio-data") pod "bc1b720f-20bc-4c1b-b075-d7b3df0d5253" (UID: "bc1b720f-20bc-4c1b-b075-d7b3df0d5253"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:31:14.720501 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.720487    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bc1b720f-20bc-4c1b-b075-d7b3df0d5253" (UID: "bc1b720f-20bc-4c1b-b075-d7b3df0d5253"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:31:14.722119 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.722078    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-token" (OuterVolumeSpecName: "istio-token") pod "bc1b720f-20bc-4c1b-b075-d7b3df0d5253" (UID: "bc1b720f-20bc-4c1b-b075-d7b3df0d5253"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:31:14.722370 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.722350    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-kube-api-access-8xn7v" (OuterVolumeSpecName: "kube-api-access-8xn7v") pod "bc1b720f-20bc-4c1b-b075-d7b3df0d5253" (UID: "bc1b720f-20bc-4c1b-b075-d7b3df0d5253"). InnerVolumeSpecName "kube-api-access-8xn7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:31:14.722595 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.722483    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bc1b720f-20bc-4c1b-b075-d7b3df0d5253" (UID: "bc1b720f-20bc-4c1b-b075-d7b3df0d5253"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:31:14.722595 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.722562    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bc1b720f-20bc-4c1b-b075-d7b3df0d5253" (UID: "bc1b720f-20bc-4c1b-b075-d7b3df0d5253"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 21:31:14.821255 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.821218    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-data\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:31:14.821255 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.821253    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-workload-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:31:14.821255 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.821263    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istiod-ca-cert\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:31:14.821491 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.821271    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-envoy\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:31:14.821491 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.821279    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-podinfo\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:31:14.821491 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.821287    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xn7v\" (UniqueName: \"kubernetes.io/projected/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-kube-api-access-8xn7v\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:31:14.821491 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:14.821297    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1b720f-20bc-4c1b-b075-d7b3df0d5253-istio-token\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:31:15.152083 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:15.152001    2559 generic.go:356] "Generic (PLEG): container finished" podID="bc1b720f-20bc-4c1b-b075-d7b3df0d5253" containerID="7fe9c47a653ff235260d058e7b06374d79513caba8f43e9b490d54c65bbc6c2c" exitCode=0
Jul 06 21:31:15.152083 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:15.152052    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"
Jul 06 21:31:15.152083 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:15.152072    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg" event={"ID":"bc1b720f-20bc-4c1b-b075-d7b3df0d5253","Type":"ContainerDied","Data":"7fe9c47a653ff235260d058e7b06374d79513caba8f43e9b490d54c65bbc6c2c"}
Jul 06 21:31:15.152393 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:15.152120    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg" event={"ID":"bc1b720f-20bc-4c1b-b075-d7b3df0d5253","Type":"ContainerDied","Data":"32b12de950b5f291ff91cac1fe86c3506eecb2cb6ceb7c435422bc6bd1fdd564"}
Jul 06 21:31:15.152393 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:15.152142    2559 scope.go:117] "RemoveContainer" containerID="7fe9c47a653ff235260d058e7b06374d79513caba8f43e9b490d54c65bbc6c2c"
Jul 06 21:31:15.153785 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:15.153760    2559 generic.go:356] "Generic (PLEG): container finished" podID="047c1603-6763-4cb1-8c0f-8094f55a94b7" containerID="cf7a5bd753fd3534a0f5324784e67821b78f5444d0487b8fc2e94e5c9f81a454" exitCode=0
Jul 06 21:31:15.153890 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:15.153825    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44" event={"ID":"047c1603-6763-4cb1-8c0f-8094f55a94b7","Type":"ContainerDied","Data":"cf7a5bd753fd3534a0f5324784e67821b78f5444d0487b8fc2e94e5c9f81a454"}
Jul 06 21:31:15.154179 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:15.154165    2559 scope.go:117] "RemoveContainer" containerID="cf7a5bd753fd3534a0f5324784e67821b78f5444d0487b8fc2e94e5c9f81a454"
Jul 06 21:31:15.155260 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:15.155234    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:31:15.155333 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:15.155313    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:31:15.155376 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:15.155360    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 06 21:31:15.164033 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:15.164014    2559 scope.go:117] "RemoveContainer" containerID="7fe9c47a653ff235260d058e7b06374d79513caba8f43e9b490d54c65bbc6c2c"
Jul 06 21:31:15.164284 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:31:15.164267    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe9c47a653ff235260d058e7b06374d79513caba8f43e9b490d54c65bbc6c2c\": container with ID starting with 7fe9c47a653ff235260d058e7b06374d79513caba8f43e9b490d54c65bbc6c2c not found: ID does not exist" containerID="7fe9c47a653ff235260d058e7b06374d79513caba8f43e9b490d54c65bbc6c2c"
Jul 06 21:31:15.164354 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:15.164292    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe9c47a653ff235260d058e7b06374d79513caba8f43e9b490d54c65bbc6c2c"} err="failed to get container status \"7fe9c47a653ff235260d058e7b06374d79513caba8f43e9b490d54c65bbc6c2c\": rpc error: code = NotFound desc = could not find container \"7fe9c47a653ff235260d058e7b06374d79513caba8f43e9b490d54c65bbc6c2c\": container with ID starting with 7fe9c47a653ff235260d058e7b06374d79513caba8f43e9b490d54c65bbc6c2c not found: ID does not exist"
Jul 06 21:31:15.191469 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:15.191444    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"]
Jul 06 21:31:15.195832 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:15.195807    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg"]
Jul 06 21:31:15.657727 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:15.657690    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc1b720f-20bc-4c1b-b075-d7b3df0d5253" path="/var/lib/kubelet/pods/bc1b720f-20bc-4c1b-b075-d7b3df0d5253/volumes"
Jul 06 21:31:16.160908 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:16.160869    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44" event={"ID":"047c1603-6763-4cb1-8c0f-8094f55a94b7","Type":"ContainerStarted","Data":"03ec33134814712e678133db6cf048088696d427be59575e6b6a6a02f078b503"}
Jul 06 21:31:16.175211 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:16.175166    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:16.175477 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:16.175448    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44" podUID="047c1603-6763-4cb1-8c0f-8094f55a94b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 06 21:31:17.175336 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:17.175296    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44" podUID="047c1603-6763-4cb1-8c0f-8094f55a94b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 06 21:31:18.176003 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:18.175966    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44" podUID="047c1603-6763-4cb1-8c0f-8094f55a94b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 06 21:31:19.175361 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:19.175325    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44" podUID="047c1603-6763-4cb1-8c0f-8094f55a94b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 06 21:31:20.176062 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:20.176014    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44" podUID="047c1603-6763-4cb1-8c0f-8094f55a94b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 06 21:31:21.098490 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:21.098444    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:21.179355 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:21.179322    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:31:21.186834 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:31:21.186809    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44"
Jul 06 21:36:44.951810 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:36:44.951781    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv_029c2f6a-b869-4b84-b178-3b7dc3eef6fe/main/0.log"
Jul 06 21:54:16.407000 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.406969    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"]
Jul 06 21:54:16.409400 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.407340    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc1b720f-20bc-4c1b-b075-d7b3df0d5253" containerName="istio-proxy"
Jul 06 21:54:16.409400 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.407350    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc1b720f-20bc-4c1b-b075-d7b3df0d5253" containerName="istio-proxy"
Jul 06 21:54:16.409400 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.407412    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc1b720f-20bc-4c1b-b075-d7b3df0d5253" containerName="istio-proxy"
Jul 06 21:54:16.410365 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.410350    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:16.412536 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.412520    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 06 21:54:16.433163 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.433143    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"]
Jul 06 21:54:16.479074 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.479029    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-vfth6\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:16.479074 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.479077    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c2945e-850d-4251-8388-dcae61f558db-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-vfth6\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:16.479286 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.479139    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-vfth6\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:16.479286 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.479171    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-vfth6\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:16.479286 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.479240    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-vfth6\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:16.479384 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.479305    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28jcl\" (UniqueName: \"kubernetes.io/projected/d8c2945e-850d-4251-8388-dcae61f558db-kube-api-access-28jcl\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-vfth6\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:16.579874 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.579826    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-vfth6\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:16.580069 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.579930    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28jcl\" (UniqueName: \"kubernetes.io/projected/d8c2945e-850d-4251-8388-dcae61f558db-kube-api-access-28jcl\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-vfth6\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:16.580069 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.579982    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-vfth6\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:16.580069 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.580015    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c2945e-850d-4251-8388-dcae61f558db-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-vfth6\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:16.580069 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.580055    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-vfth6\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:16.580299 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.580077    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-vfth6\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:16.580543 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.580517    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-vfth6\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:16.580766 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.580696    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-vfth6\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:16.581110 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.581086    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-vfth6\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:16.583400 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.583359    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-vfth6\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:16.583806 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.583779    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c2945e-850d-4251-8388-dcae61f558db-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-vfth6\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:16.588336 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.588307    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28jcl\" (UniqueName: \"kubernetes.io/projected/d8c2945e-850d-4251-8388-dcae61f558db-kube-api-access-28jcl\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-vfth6\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:16.720705 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.720602    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:16.859716 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:16.859685    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"]
Jul 06 21:54:16.861387 ip-10-0-134-202 kubenswrapper[2559]: W0706 21:54:16.861358    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c2945e_850d_4251_8388_dcae61f558db.slice/crio-98272055fb4d267eeb14e9c263c12bcdab090aa2f3e689c262ba503ea1a0e40f WatchSource:0}: Error finding container 98272055fb4d267eeb14e9c263c12bcdab090aa2f3e689c262ba503ea1a0e40f: Status 404 returned error can't find the container with id 98272055fb4d267eeb14e9c263c12bcdab090aa2f3e689c262ba503ea1a0e40f
Jul 06 21:54:17.592658 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:17.592602    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6" event={"ID":"d8c2945e-850d-4251-8388-dcae61f558db","Type":"ContainerStarted","Data":"f22a8d9eeb9ee159d8f4fbb874820c2ec532a7fdfb27a0b2eb671e65540abf6d"}
Jul 06 21:54:17.593017 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:17.592679    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6" event={"ID":"d8c2945e-850d-4251-8388-dcae61f558db","Type":"ContainerStarted","Data":"98272055fb4d267eeb14e9c263c12bcdab090aa2f3e689c262ba503ea1a0e40f"}
Jul 06 21:54:17.632187 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:17.632130    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6" podStartSLOduration=1.632113951 podStartE2EDuration="1.632113951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:54:17.630779385 +0000 UTC m=+3906.465666769" watchObservedRunningTime="2026-07-06 21:54:17.632113951 +0000 UTC m=+3906.467001335"
Jul 06 21:54:17.721071 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:17.721039    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:26.721685 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:26.721572    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:26.734207 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:26.734181    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:27.650938 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:27.650904    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:54:58.011711 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:54:58.011672    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"]
Jul 06 21:55:13.032423 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.032373    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6" podUID="d8c2945e-850d-4251-8388-dcae61f558db" containerName="main" containerID="cri-o://f22a8d9eeb9ee159d8f4fbb874820c2ec532a7fdfb27a0b2eb671e65540abf6d" gracePeriod=45
Jul 06 21:55:13.291905 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.291882    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:55:13.399020 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.398983    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-model-cache\") pod \"d8c2945e-850d-4251-8388-dcae61f558db\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") "
Jul 06 21:55:13.399020 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.399027    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-home\") pod \"d8c2945e-850d-4251-8388-dcae61f558db\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") "
Jul 06 21:55:13.399281 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.399051    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-tmp-dir\") pod \"d8c2945e-850d-4251-8388-dcae61f558db\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") "
Jul 06 21:55:13.399281 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.399083    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28jcl\" (UniqueName: \"kubernetes.io/projected/d8c2945e-850d-4251-8388-dcae61f558db-kube-api-access-28jcl\") pod \"d8c2945e-850d-4251-8388-dcae61f558db\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") "
Jul 06 21:55:13.399281 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.399133    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c2945e-850d-4251-8388-dcae61f558db-tls-certs\") pod \"d8c2945e-850d-4251-8388-dcae61f558db\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") "
Jul 06 21:55:13.399281 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.399162    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-dshm\") pod \"d8c2945e-850d-4251-8388-dcae61f558db\" (UID: \"d8c2945e-850d-4251-8388-dcae61f558db\") "
Jul 06 21:55:13.399470 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.399305    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-model-cache" (OuterVolumeSpecName: "model-cache") pod "d8c2945e-850d-4251-8388-dcae61f558db" (UID: "d8c2945e-850d-4251-8388-dcae61f558db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:55:13.399470 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.399340    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d8c2945e-850d-4251-8388-dcae61f558db" (UID: "d8c2945e-850d-4251-8388-dcae61f558db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:55:13.399470 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.399362    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-home" (OuterVolumeSpecName: "home") pod "d8c2945e-850d-4251-8388-dcae61f558db" (UID: "d8c2945e-850d-4251-8388-dcae61f558db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:55:13.399652 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.399516    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-tmp-dir\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:55:13.399652 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.399530    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-model-cache\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:55:13.399652 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.399538    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-home\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:55:13.401285 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.401253    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-dshm" (OuterVolumeSpecName: "dshm") pod "d8c2945e-850d-4251-8388-dcae61f558db" (UID: "d8c2945e-850d-4251-8388-dcae61f558db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:55:13.401285 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.401266    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c2945e-850d-4251-8388-dcae61f558db-kube-api-access-28jcl" (OuterVolumeSpecName: "kube-api-access-28jcl") pod "d8c2945e-850d-4251-8388-dcae61f558db" (UID: "d8c2945e-850d-4251-8388-dcae61f558db"). InnerVolumeSpecName "kube-api-access-28jcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:55:13.401434 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.401287    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c2945e-850d-4251-8388-dcae61f558db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d8c2945e-850d-4251-8388-dcae61f558db" (UID: "d8c2945e-850d-4251-8388-dcae61f558db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:55:13.500879 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.500844    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c2945e-850d-4251-8388-dcae61f558db-dshm\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:55:13.501046 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.500873    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28jcl\" (UniqueName: \"kubernetes.io/projected/d8c2945e-850d-4251-8388-dcae61f558db-kube-api-access-28jcl\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:55:13.501046 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.500908    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c2945e-850d-4251-8388-dcae61f558db-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:55:13.855548 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.855514    2559 generic.go:356] "Generic (PLEG): container finished" podID="d8c2945e-850d-4251-8388-dcae61f558db" containerID="f22a8d9eeb9ee159d8f4fbb874820c2ec532a7fdfb27a0b2eb671e65540abf6d" exitCode=0
Jul 06 21:55:13.855724 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.855557    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"
Jul 06 21:55:13.855724 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.855563    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6" event={"ID":"d8c2945e-850d-4251-8388-dcae61f558db","Type":"ContainerDied","Data":"f22a8d9eeb9ee159d8f4fbb874820c2ec532a7fdfb27a0b2eb671e65540abf6d"}
Jul 06 21:55:13.855724 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.855635    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6" event={"ID":"d8c2945e-850d-4251-8388-dcae61f558db","Type":"ContainerDied","Data":"98272055fb4d267eeb14e9c263c12bcdab090aa2f3e689c262ba503ea1a0e40f"}
Jul 06 21:55:13.855724 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.855639    2559 scope.go:117] "RemoveContainer" containerID="f22a8d9eeb9ee159d8f4fbb874820c2ec532a7fdfb27a0b2eb671e65540abf6d"
Jul 06 21:55:13.863984 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.863967    2559 scope.go:117] "RemoveContainer" containerID="f22a8d9eeb9ee159d8f4fbb874820c2ec532a7fdfb27a0b2eb671e65540abf6d"
Jul 06 21:55:13.864220 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:55:13.864203    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f22a8d9eeb9ee159d8f4fbb874820c2ec532a7fdfb27a0b2eb671e65540abf6d\": container with ID starting with f22a8d9eeb9ee159d8f4fbb874820c2ec532a7fdfb27a0b2eb671e65540abf6d not found: ID does not exist" containerID="f22a8d9eeb9ee159d8f4fbb874820c2ec532a7fdfb27a0b2eb671e65540abf6d"
Jul 06 21:55:13.864268 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.864229    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f22a8d9eeb9ee159d8f4fbb874820c2ec532a7fdfb27a0b2eb671e65540abf6d"} err="failed to get container status \"f22a8d9eeb9ee159d8f4fbb874820c2ec532a7fdfb27a0b2eb671e65540abf6d\": rpc error: code = NotFound desc = could not find container \"f22a8d9eeb9ee159d8f4fbb874820c2ec532a7fdfb27a0b2eb671e65540abf6d\": container with ID starting with f22a8d9eeb9ee159d8f4fbb874820c2ec532a7fdfb27a0b2eb671e65540abf6d not found: ID does not exist"
Jul 06 21:55:13.872469 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.872444    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"]
Jul 06 21:55:13.874676 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:13.874649    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-vfth6"]
Jul 06 21:55:15.657778 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:15.657747    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8c2945e-850d-4251-8388-dcae61f558db" path="/var/lib/kubelet/pods/d8c2945e-850d-4251-8388-dcae61f558db/volumes"
Jul 06 21:55:35.305685 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.305654    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"]
Jul 06 21:55:35.306115 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.306028    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8c2945e-850d-4251-8388-dcae61f558db" containerName="main"
Jul 06 21:55:35.306115 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.306039    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8c2945e-850d-4251-8388-dcae61f558db" containerName="main"
Jul 06 21:55:35.306115 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.306091    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8c2945e-850d-4251-8388-dcae61f558db" containerName="main"
Jul 06 21:55:35.309067 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.309050    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:35.311426 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.311403    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 06 21:55:35.318622 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.318584    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"]
Jul 06 21:55:35.494227 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.494191    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:35.494410 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.494236    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a3c2500-012a-4748-8f45-436e5899cd3e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:35.494410 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.494316    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:35.494410 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.494359    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:35.494410 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.494396    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:35.494564 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.494445    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65rxr\" (UniqueName: \"kubernetes.io/projected/1a3c2500-012a-4748-8f45-436e5899cd3e-kube-api-access-65rxr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:35.595716 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.595632    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:35.595716 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.595686    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:35.595893 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.595797    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:35.595893 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.595867    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65rxr\" (UniqueName: \"kubernetes.io/projected/1a3c2500-012a-4748-8f45-436e5899cd3e-kube-api-access-65rxr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:35.595974 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.595957    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:35.596007 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.595997    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a3c2500-012a-4748-8f45-436e5899cd3e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:35.596053 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.596039    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:35.596093 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.596078    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:35.596307 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.596276    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:35.597923 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.597896    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:35.598208 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.598190    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a3c2500-012a-4748-8f45-436e5899cd3e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:35.606703 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.606681    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65rxr\" (UniqueName: \"kubernetes.io/projected/1a3c2500-012a-4748-8f45-436e5899cd3e-kube-api-access-65rxr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:35.619519 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.619489    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:35.758001 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.757959    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"]
Jul 06 21:55:35.761563 ip-10-0-134-202 kubenswrapper[2559]: W0706 21:55:35.761537    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3c2500_012a_4748_8f45_436e5899cd3e.slice/crio-839f5a8064abf830f8779e61f1a2d819dd26aa96045cc49a4af326db3a691d2f WatchSource:0}: Error finding container 839f5a8064abf830f8779e61f1a2d819dd26aa96045cc49a4af326db3a691d2f: Status 404 returned error can't find the container with id 839f5a8064abf830f8779e61f1a2d819dd26aa96045cc49a4af326db3a691d2f
Jul 06 21:55:35.958643 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.958529    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb" event={"ID":"1a3c2500-012a-4748-8f45-436e5899cd3e","Type":"ContainerStarted","Data":"9a757d35c91c4bab53c90c211fdb159a43a0bfda4ade41f6cdc9e4b6d1fcb573"}
Jul 06 21:55:35.958643 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.958596    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb" event={"ID":"1a3c2500-012a-4748-8f45-436e5899cd3e","Type":"ContainerStarted","Data":"839f5a8064abf830f8779e61f1a2d819dd26aa96045cc49a4af326db3a691d2f"}
Jul 06 21:55:35.980163 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:35.980116    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb" podStartSLOduration=0.980102789 podStartE2EDuration="980.102789ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:55:35.977518337 +0000 UTC m=+3984.812405718" watchObservedRunningTime="2026-07-06 21:55:35.980102789 +0000 UTC m=+3984.814990153"
Jul 06 21:55:36.620418 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:36.620387    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:45.620592 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:45.620555    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:45.632962 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:45.632935    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:55:46.020089 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:55:46.020003    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:56:34.970996 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:34.970901    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"]
Jul 06 21:56:49.996886 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:49.996842    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb" podUID="1a3c2500-012a-4748-8f45-436e5899cd3e" containerName="main" containerID="cri-o://9a757d35c91c4bab53c90c211fdb159a43a0bfda4ade41f6cdc9e4b6d1fcb573" gracePeriod=45
Jul 06 21:56:50.009912 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.009876    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb" podUID="1a3c2500-012a-4748-8f45-436e5899cd3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 06 21:56:50.249728 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.249665    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:56:50.315873 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.315843    2559 generic.go:356] "Generic (PLEG): container finished" podID="1a3c2500-012a-4748-8f45-436e5899cd3e" containerID="9a757d35c91c4bab53c90c211fdb159a43a0bfda4ade41f6cdc9e4b6d1fcb573" exitCode=0
Jul 06 21:56:50.316037 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.315889    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"
Jul 06 21:56:50.316037 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.315933    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb" event={"ID":"1a3c2500-012a-4748-8f45-436e5899cd3e","Type":"ContainerDied","Data":"9a757d35c91c4bab53c90c211fdb159a43a0bfda4ade41f6cdc9e4b6d1fcb573"}
Jul 06 21:56:50.316037 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.315983    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb" event={"ID":"1a3c2500-012a-4748-8f45-436e5899cd3e","Type":"ContainerDied","Data":"839f5a8064abf830f8779e61f1a2d819dd26aa96045cc49a4af326db3a691d2f"}
Jul 06 21:56:50.316037 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.316004    2559 scope.go:117] "RemoveContainer" containerID="9a757d35c91c4bab53c90c211fdb159a43a0bfda4ade41f6cdc9e4b6d1fcb573"
Jul 06 21:56:50.324248 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.324231    2559 scope.go:117] "RemoveContainer" containerID="9a757d35c91c4bab53c90c211fdb159a43a0bfda4ade41f6cdc9e4b6d1fcb573"
Jul 06 21:56:50.324496 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:56:50.324480    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a757d35c91c4bab53c90c211fdb159a43a0bfda4ade41f6cdc9e4b6d1fcb573\": container with ID starting with 9a757d35c91c4bab53c90c211fdb159a43a0bfda4ade41f6cdc9e4b6d1fcb573 not found: ID does not exist" containerID="9a757d35c91c4bab53c90c211fdb159a43a0bfda4ade41f6cdc9e4b6d1fcb573"
Jul 06 21:56:50.324557 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.324504    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a757d35c91c4bab53c90c211fdb159a43a0bfda4ade41f6cdc9e4b6d1fcb573"} err="failed to get container status \"9a757d35c91c4bab53c90c211fdb159a43a0bfda4ade41f6cdc9e4b6d1fcb573\": rpc error: code = NotFound desc = could not find container \"9a757d35c91c4bab53c90c211fdb159a43a0bfda4ade41f6cdc9e4b6d1fcb573\": container with ID starting with 9a757d35c91c4bab53c90c211fdb159a43a0bfda4ade41f6cdc9e4b6d1fcb573 not found: ID does not exist"
Jul 06 21:56:50.362871 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.362843    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a3c2500-012a-4748-8f45-436e5899cd3e-tls-certs\") pod \"1a3c2500-012a-4748-8f45-436e5899cd3e\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") "
Jul 06 21:56:50.362980 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.362890    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-model-cache\") pod \"1a3c2500-012a-4748-8f45-436e5899cd3e\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") "
Jul 06 21:56:50.362980 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.362907    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-tmp-dir\") pod \"1a3c2500-012a-4748-8f45-436e5899cd3e\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") "
Jul 06 21:56:50.362980 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.362923    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-dshm\") pod \"1a3c2500-012a-4748-8f45-436e5899cd3e\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") "
Jul 06 21:56:50.362980 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.362940    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-home\") pod \"1a3c2500-012a-4748-8f45-436e5899cd3e\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") "
Jul 06 21:56:50.363143 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.363041    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65rxr\" (UniqueName: \"kubernetes.io/projected/1a3c2500-012a-4748-8f45-436e5899cd3e-kube-api-access-65rxr\") pod \"1a3c2500-012a-4748-8f45-436e5899cd3e\" (UID: \"1a3c2500-012a-4748-8f45-436e5899cd3e\") "
Jul 06 21:56:50.363203 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.363154    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a3c2500-012a-4748-8f45-436e5899cd3e" (UID: "1a3c2500-012a-4748-8f45-436e5899cd3e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:56:50.363268 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.363200    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a3c2500-012a-4748-8f45-436e5899cd3e" (UID: "1a3c2500-012a-4748-8f45-436e5899cd3e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:56:50.363384 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.363364    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-model-cache\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:56:50.363452 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.363393    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-tmp-dir\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:56:50.363452 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.363364    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-home" (OuterVolumeSpecName: "home") pod "1a3c2500-012a-4748-8f45-436e5899cd3e" (UID: "1a3c2500-012a-4748-8f45-436e5899cd3e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:56:50.365060 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.365036    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-dshm" (OuterVolumeSpecName: "dshm") pod "1a3c2500-012a-4748-8f45-436e5899cd3e" (UID: "1a3c2500-012a-4748-8f45-436e5899cd3e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:56:50.365142 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.365064    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a3c2500-012a-4748-8f45-436e5899cd3e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a3c2500-012a-4748-8f45-436e5899cd3e" (UID: "1a3c2500-012a-4748-8f45-436e5899cd3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:56:50.365198 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.365153    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a3c2500-012a-4748-8f45-436e5899cd3e-kube-api-access-65rxr" (OuterVolumeSpecName: "kube-api-access-65rxr") pod "1a3c2500-012a-4748-8f45-436e5899cd3e" (UID: "1a3c2500-012a-4748-8f45-436e5899cd3e"). InnerVolumeSpecName "kube-api-access-65rxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:56:50.464362 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.464333    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-dshm\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:56:50.464362 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.464357    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a3c2500-012a-4748-8f45-436e5899cd3e-home\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:56:50.464362 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.464367    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65rxr\" (UniqueName: \"kubernetes.io/projected/1a3c2500-012a-4748-8f45-436e5899cd3e-kube-api-access-65rxr\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:56:50.464643 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.464376    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a3c2500-012a-4748-8f45-436e5899cd3e-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:56:50.637568 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.637542    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"]
Jul 06 21:56:50.641667 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:50.641645    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvxrb"]
Jul 06 21:56:51.658222 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:56:51.658181    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a3c2500-012a-4748-8f45-436e5899cd3e" path="/var/lib/kubelet/pods/1a3c2500-012a-4748-8f45-436e5899cd3e/volumes"
Jul 06 21:57:16.212435 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.212404    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"]
Jul 06 21:57:16.212863 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.212775    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a3c2500-012a-4748-8f45-436e5899cd3e" containerName="main"
Jul 06 21:57:16.212863 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.212787    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a3c2500-012a-4748-8f45-436e5899cd3e" containerName="main"
Jul 06 21:57:16.212942 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.212867    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a3c2500-012a-4748-8f45-436e5899cd3e" containerName="main"
Jul 06 21:57:16.217032 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.217015    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:16.219158 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.219141    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 06 21:57:16.227438 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.227412    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"]
Jul 06 21:57:16.286358 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.286323    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:16.286549 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.286376    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/146df702-52b7-4ea3-a1e6-91ad11d31ab7-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:16.286549 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.286420    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:16.286549 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.286471    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7bnv\" (UniqueName: \"kubernetes.io/projected/146df702-52b7-4ea3-a1e6-91ad11d31ab7-kube-api-access-m7bnv\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:16.286549 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.286521    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:16.286755 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.286629    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:16.387701 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.387658    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7bnv\" (UniqueName: \"kubernetes.io/projected/146df702-52b7-4ea3-a1e6-91ad11d31ab7-kube-api-access-m7bnv\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:16.387897 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.387730    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:16.387897 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.387772    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:16.387897 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.387831    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:16.387897 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.387878    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/146df702-52b7-4ea3-a1e6-91ad11d31ab7-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:16.388074 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.387904    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:16.388242 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.388218    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:16.388317 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.388255    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:16.388372 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.388315    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:16.390028 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.390001    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:16.390339 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.390321    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/146df702-52b7-4ea3-a1e6-91ad11d31ab7-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:16.396024 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.396000    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7bnv\" (UniqueName: \"kubernetes.io/projected/146df702-52b7-4ea3-a1e6-91ad11d31ab7-kube-api-access-m7bnv\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:16.527938 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.527852    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:16.668274 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:16.668250    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"]
Jul 06 21:57:16.670291 ip-10-0-134-202 kubenswrapper[2559]: W0706 21:57:16.670268    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146df702_52b7_4ea3_a1e6_91ad11d31ab7.slice/crio-45d86e6d8da442091873fcd95bc6b6860ac64cac118f211767afe1244df0d325 WatchSource:0}: Error finding container 45d86e6d8da442091873fcd95bc6b6860ac64cac118f211767afe1244df0d325: Status 404 returned error can't find the container with id 45d86e6d8da442091873fcd95bc6b6860ac64cac118f211767afe1244df0d325
Jul 06 21:57:17.442947 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:17.442908    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w" event={"ID":"146df702-52b7-4ea3-a1e6-91ad11d31ab7","Type":"ContainerStarted","Data":"5f57e2aed9c0e9ff2a9beba49656604ac4a6bbc18e5842b69585927cdbd362ab"}
Jul 06 21:57:17.443355 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:17.442970    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w" event={"ID":"146df702-52b7-4ea3-a1e6-91ad11d31ab7","Type":"ContainerStarted","Data":"45d86e6d8da442091873fcd95bc6b6860ac64cac118f211767afe1244df0d325"}
Jul 06 21:57:17.528206 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:17.528173    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:26.528086 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:26.528040    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:26.540642 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:26.540594    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:57:26.592835 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:26.592783    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w" podStartSLOduration=10.592764536 podStartE2EDuration="10.592764536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:57:17.464381267 +0000 UTC m=+4086.299268633" watchObservedRunningTime="2026-07-06 21:57:26.592764536 +0000 UTC m=+4095.427651902"
Jul 06 21:57:27.504719 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:57:27.504687    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:58:11.501037 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:11.500999    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"]
Jul 06 21:58:26.521932 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:26.521868    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w" podUID="146df702-52b7-4ea3-a1e6-91ad11d31ab7" containerName="main" containerID="cri-o://5f57e2aed9c0e9ff2a9beba49656604ac4a6bbc18e5842b69585927cdbd362ab" gracePeriod=45
Jul 06 21:58:26.773504 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:26.773469    2559 generic.go:356] "Generic (PLEG): container finished" podID="146df702-52b7-4ea3-a1e6-91ad11d31ab7" containerID="5f57e2aed9c0e9ff2a9beba49656604ac4a6bbc18e5842b69585927cdbd362ab" exitCode=0
Jul 06 21:58:26.773670 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:26.773534    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w" event={"ID":"146df702-52b7-4ea3-a1e6-91ad11d31ab7","Type":"ContainerDied","Data":"5f57e2aed9c0e9ff2a9beba49656604ac4a6bbc18e5842b69585927cdbd362ab"}
Jul 06 21:58:26.773670 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:26.773584    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w" event={"ID":"146df702-52b7-4ea3-a1e6-91ad11d31ab7","Type":"ContainerDied","Data":"45d86e6d8da442091873fcd95bc6b6860ac64cac118f211767afe1244df0d325"}
Jul 06 21:58:26.773670 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:26.773624    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45d86e6d8da442091873fcd95bc6b6860ac64cac118f211767afe1244df0d325"
Jul 06 21:58:26.775940 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:26.775919    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:58:26.951685 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:26.951576    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-home\") pod \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") "
Jul 06 21:58:26.951685 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:26.951658    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-dshm\") pod \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") "
Jul 06 21:58:26.951685 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:26.951681    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/146df702-52b7-4ea3-a1e6-91ad11d31ab7-tls-certs\") pod \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") "
Jul 06 21:58:26.951955 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:26.951743    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7bnv\" (UniqueName: \"kubernetes.io/projected/146df702-52b7-4ea3-a1e6-91ad11d31ab7-kube-api-access-m7bnv\") pod \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") "
Jul 06 21:58:26.951955 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:26.951770    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-tmp-dir\") pod \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") "
Jul 06 21:58:26.951955 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:26.951806    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-model-cache\") pod \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\" (UID: \"146df702-52b7-4ea3-a1e6-91ad11d31ab7\") "
Jul 06 21:58:26.951955 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:26.951882    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-home" (OuterVolumeSpecName: "home") pod "146df702-52b7-4ea3-a1e6-91ad11d31ab7" (UID: "146df702-52b7-4ea3-a1e6-91ad11d31ab7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:26.952214 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:26.952093    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "146df702-52b7-4ea3-a1e6-91ad11d31ab7" (UID: "146df702-52b7-4ea3-a1e6-91ad11d31ab7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:26.952214 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:26.952098    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-model-cache" (OuterVolumeSpecName: "model-cache") pod "146df702-52b7-4ea3-a1e6-91ad11d31ab7" (UID: "146df702-52b7-4ea3-a1e6-91ad11d31ab7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:26.952214 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:26.952156    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-home\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:58:26.953859 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:26.953831    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146df702-52b7-4ea3-a1e6-91ad11d31ab7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "146df702-52b7-4ea3-a1e6-91ad11d31ab7" (UID: "146df702-52b7-4ea3-a1e6-91ad11d31ab7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:58:26.953971 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:26.953867    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-dshm" (OuterVolumeSpecName: "dshm") pod "146df702-52b7-4ea3-a1e6-91ad11d31ab7" (UID: "146df702-52b7-4ea3-a1e6-91ad11d31ab7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:58:26.953971 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:26.953894    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146df702-52b7-4ea3-a1e6-91ad11d31ab7-kube-api-access-m7bnv" (OuterVolumeSpecName: "kube-api-access-m7bnv") pod "146df702-52b7-4ea3-a1e6-91ad11d31ab7" (UID: "146df702-52b7-4ea3-a1e6-91ad11d31ab7"). InnerVolumeSpecName "kube-api-access-m7bnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:58:27.053003 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:27.052969    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-dshm\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:58:27.053003 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:27.052997    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/146df702-52b7-4ea3-a1e6-91ad11d31ab7-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:58:27.053003 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:27.053007    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7bnv\" (UniqueName: \"kubernetes.io/projected/146df702-52b7-4ea3-a1e6-91ad11d31ab7-kube-api-access-m7bnv\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:58:27.053229 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:27.053016    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-tmp-dir\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:58:27.053229 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:27.053026    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/146df702-52b7-4ea3-a1e6-91ad11d31ab7-model-cache\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:58:27.778567 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:27.778529    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"
Jul 06 21:58:27.795558 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:27.795526    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"]
Jul 06 21:58:27.800685 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:27.800663    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cg2w"]
Jul 06 21:58:29.658099 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:29.658067    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="146df702-52b7-4ea3-a1e6-91ad11d31ab7" path="/var/lib/kubelet/pods/146df702-52b7-4ea3-a1e6-91ad11d31ab7/volumes"
Jul 06 21:58:52.103299 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.103263    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"]
Jul 06 21:58:52.103787 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.103648    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="146df702-52b7-4ea3-a1e6-91ad11d31ab7" containerName="main"
Jul 06 21:58:52.103787 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.103659    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="146df702-52b7-4ea3-a1e6-91ad11d31ab7" containerName="main"
Jul 06 21:58:52.103787 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.103745    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="146df702-52b7-4ea3-a1e6-91ad11d31ab7" containerName="main"
Jul 06 21:58:52.106958 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.106942    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:58:52.109087 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.109065    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 06 21:58:52.117855 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.117833    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"]
Jul 06 21:58:52.180954 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.180916    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:58:52.181153 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.180986    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3e5a988-84fe-4482-b0f6-7121b9b349a8-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:58:52.181153 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.181007    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zjqj\" (UniqueName: \"kubernetes.io/projected/c3e5a988-84fe-4482-b0f6-7121b9b349a8-kube-api-access-9zjqj\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:58:52.181153 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.181055    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:58:52.181153 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.181134    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:58:52.181317 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.181162    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:58:52.282419 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.282384    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:58:52.282659 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.282434    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:58:52.282659 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.282463    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:58:52.282659 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.282515    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:58:52.282659 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.282578    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3e5a988-84fe-4482-b0f6-7121b9b349a8-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:58:52.282659 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.282604    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zjqj\" (UniqueName: \"kubernetes.io/projected/c3e5a988-84fe-4482-b0f6-7121b9b349a8-kube-api-access-9zjqj\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:58:52.283070 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.282949    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:58:52.283070 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.282962    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:58:52.283070 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.283017    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:58:52.285673 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.285648    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:58:52.285842 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.285782    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3e5a988-84fe-4482-b0f6-7121b9b349a8-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:58:52.290736 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.290709    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zjqj\" (UniqueName: \"kubernetes.io/projected/c3e5a988-84fe-4482-b0f6-7121b9b349a8-kube-api-access-9zjqj\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:58:52.417923 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.417824    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:58:52.444751 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.444721    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk"]
Jul 06 21:58:52.448344 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.448320    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk"
Jul 06 21:58:52.450933 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.450910    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-jjt7d\""
Jul 06 21:58:52.460206 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.460177    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk"]
Jul 06 21:58:52.580053 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.580025    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"]
Jul 06 21:58:52.583549 ip-10-0-134-202 kubenswrapper[2559]: W0706 21:58:52.583525    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e5a988_84fe_4482_b0f6_7121b9b349a8.slice/crio-a783d100619946858464d6d1272f259bb1f337af38af8a75e667e89919e90add WatchSource:0}: Error finding container a783d100619946858464d6d1272f259bb1f337af38af8a75e667e89919e90add: Status 404 returned error can't find the container with id a783d100619946858464d6d1272f259bb1f337af38af8a75e667e89919e90add
Jul 06 21:58:52.585157 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.585135    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a84965d-efea-4843-b5b2-d35f84f66c77-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk\" (UID: \"2a84965d-efea-4843-b5b2-d35f84f66c77\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk"
Jul 06 21:58:52.585262 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.585209    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vz2n\" (UniqueName: \"kubernetes.io/projected/2a84965d-efea-4843-b5b2-d35f84f66c77-kube-api-access-2vz2n\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk\" (UID: \"2a84965d-efea-4843-b5b2-d35f84f66c77\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk"
Jul 06 21:58:52.585326 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.585281    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a84965d-efea-4843-b5b2-d35f84f66c77-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk\" (UID: \"2a84965d-efea-4843-b5b2-d35f84f66c77\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk"
Jul 06 21:58:52.686104 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.686025    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a84965d-efea-4843-b5b2-d35f84f66c77-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk\" (UID: \"2a84965d-efea-4843-b5b2-d35f84f66c77\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk"
Jul 06 21:58:52.686104 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.686087    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vz2n\" (UniqueName: \"kubernetes.io/projected/2a84965d-efea-4843-b5b2-d35f84f66c77-kube-api-access-2vz2n\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk\" (UID: \"2a84965d-efea-4843-b5b2-d35f84f66c77\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk"
Jul 06 21:58:52.686284 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.686150    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a84965d-efea-4843-b5b2-d35f84f66c77-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk\" (UID: \"2a84965d-efea-4843-b5b2-d35f84f66c77\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk"
Jul 06 21:58:52.686398 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.686375    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a84965d-efea-4843-b5b2-d35f84f66c77-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk\" (UID: \"2a84965d-efea-4843-b5b2-d35f84f66c77\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk"
Jul 06 21:58:52.688677 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.688658    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a84965d-efea-4843-b5b2-d35f84f66c77-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk\" (UID: \"2a84965d-efea-4843-b5b2-d35f84f66c77\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk"
Jul 06 21:58:52.694667 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.694644    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vz2n\" (UniqueName: \"kubernetes.io/projected/2a84965d-efea-4843-b5b2-d35f84f66c77-kube-api-access-2vz2n\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk\" (UID: \"2a84965d-efea-4843-b5b2-d35f84f66c77\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk"
Jul 06 21:58:52.759003 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.758969    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk"
Jul 06 21:58:52.896939 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.896895    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr" event={"ID":"c3e5a988-84fe-4482-b0f6-7121b9b349a8","Type":"ContainerStarted","Data":"b0c873f0e773ab5bc87fd1d47fb0287bd063bcd5eff5495d72e967dd294287a4"}
Jul 06 21:58:52.897121 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.897004    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr" event={"ID":"c3e5a988-84fe-4482-b0f6-7121b9b349a8","Type":"ContainerStarted","Data":"a783d100619946858464d6d1272f259bb1f337af38af8a75e667e89919e90add"}
Jul 06 21:58:52.909075 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.909053    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk"]
Jul 06 21:58:52.910983 ip-10-0-134-202 kubenswrapper[2559]: W0706 21:58:52.910956    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a84965d_efea_4843_b5b2_d35f84f66c77.slice/crio-6f59c4953aa02cc8e7c113a8c1f8895b280d248eb6162343e4f8b9889fc1916a WatchSource:0}: Error finding container 6f59c4953aa02cc8e7c113a8c1f8895b280d248eb6162343e4f8b9889fc1916a: Status 404 returned error can't find the container with id 6f59c4953aa02cc8e7c113a8c1f8895b280d248eb6162343e4f8b9889fc1916a
Jul 06 21:58:52.916160 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:52.916118    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr" podStartSLOduration=0.916104585 podStartE2EDuration="916.104585ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:58:52.915434632 +0000 UTC m=+4181.750322023" watchObservedRunningTime="2026-07-06 21:58:52.916104585 +0000 UTC m=+4181.750991950"
Jul 06 21:58:53.418482 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:53.418448    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:58:53.904273 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:53.904239    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk" event={"ID":"2a84965d-efea-4843-b5b2-d35f84f66c77","Type":"ContainerStarted","Data":"778cf2ee3aadfa4a437e87f7202aaa9ec1f96c899c9cf8757b7dc75e86327887"}
Jul 06 21:58:53.904450 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:53.904287    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk" event={"ID":"2a84965d-efea-4843-b5b2-d35f84f66c77","Type":"ContainerStarted","Data":"6f59c4953aa02cc8e7c113a8c1f8895b280d248eb6162343e4f8b9889fc1916a"}
Jul 06 21:58:53.904450 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:53.904326    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk"
Jul 06 21:58:53.926888 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:58:53.926842    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk" podStartSLOduration=1.9268268160000002 podStartE2EDuration="1.926826816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:58:53.925062608 +0000 UTC m=+4182.759949973" watchObservedRunningTime="2026-07-06 21:58:53.926826816 +0000 UTC m=+4182.761714181"
Jul 06 21:59:02.418354 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:02.418315    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:59:02.430867 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:02.430844    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:59:02.959086 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:02.959057    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:59:24.911824 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:24.911785    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk" podUID="2a84965d-efea-4843-b5b2-d35f84f66c77" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 06 21:59:33.812218 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:33.812133    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"]
Jul 06 21:59:33.820046 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:33.820016    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk"]
Jul 06 21:59:33.822148 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:33.822113    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk" podUID="2a84965d-efea-4843-b5b2-d35f84f66c77" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 06 21:59:33.829349 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:59:33.829319    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 21:59:33.829349 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:59:33.829349 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 21:59:33.829349 ip-10-0-134-202 kubenswrapper[2559]:  > containerID="778cf2ee3aadfa4a437e87f7202aaa9ec1f96c899c9cf8757b7dc75e86327887" cmd=["/bin/sleep","15"]
Jul 06 21:59:33.829665 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:33.829363    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk" message="" err=<
Jul 06 21:59:33.829665 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:59:33.829665 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 21:59:33.829665 ip-10-0-134-202 kubenswrapper[2559]:  >
Jul 06 21:59:33.829665 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:59:33.829375    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 21:59:33.829665 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 21:59:33.829665 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 21:59:33.829665 ip-10-0-134-202 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk" podUID="2a84965d-efea-4843-b5b2-d35f84f66c77" containerName="main" containerID="cri-o://778cf2ee3aadfa4a437e87f7202aaa9ec1f96c899c9cf8757b7dc75e86327887"
Jul 06 21:59:33.829665 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:33.829418    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk" podUID="2a84965d-efea-4843-b5b2-d35f84f66c77" containerName="main" containerID="cri-o://778cf2ee3aadfa4a437e87f7202aaa9ec1f96c899c9cf8757b7dc75e86327887" gracePeriod=60
Jul 06 21:59:34.085949 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:34.085920    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk"
Jul 06 21:59:34.103902 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:34.103815    2559 generic.go:356] "Generic (PLEG): container finished" podID="2a84965d-efea-4843-b5b2-d35f84f66c77" containerID="778cf2ee3aadfa4a437e87f7202aaa9ec1f96c899c9cf8757b7dc75e86327887" exitCode=0
Jul 06 21:59:34.104062 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:34.104023    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk" event={"ID":"2a84965d-efea-4843-b5b2-d35f84f66c77","Type":"ContainerDied","Data":"778cf2ee3aadfa4a437e87f7202aaa9ec1f96c899c9cf8757b7dc75e86327887"}
Jul 06 21:59:34.104132 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:34.104076    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk" event={"ID":"2a84965d-efea-4843-b5b2-d35f84f66c77","Type":"ContainerDied","Data":"6f59c4953aa02cc8e7c113a8c1f8895b280d248eb6162343e4f8b9889fc1916a"}
Jul 06 21:59:34.104132 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:34.104104    2559 scope.go:117] "RemoveContainer" containerID="778cf2ee3aadfa4a437e87f7202aaa9ec1f96c899c9cf8757b7dc75e86327887"
Jul 06 21:59:34.104250 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:34.104177    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk"
Jul 06 21:59:34.113923 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:34.113897    2559 scope.go:117] "RemoveContainer" containerID="778cf2ee3aadfa4a437e87f7202aaa9ec1f96c899c9cf8757b7dc75e86327887"
Jul 06 21:59:34.114200 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:59:34.114181    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"778cf2ee3aadfa4a437e87f7202aaa9ec1f96c899c9cf8757b7dc75e86327887\": container with ID starting with 778cf2ee3aadfa4a437e87f7202aaa9ec1f96c899c9cf8757b7dc75e86327887 not found: ID does not exist" containerID="778cf2ee3aadfa4a437e87f7202aaa9ec1f96c899c9cf8757b7dc75e86327887"
Jul 06 21:59:34.114259 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:34.114210    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"778cf2ee3aadfa4a437e87f7202aaa9ec1f96c899c9cf8757b7dc75e86327887"} err="failed to get container status \"778cf2ee3aadfa4a437e87f7202aaa9ec1f96c899c9cf8757b7dc75e86327887\": rpc error: code = NotFound desc = could not find container \"778cf2ee3aadfa4a437e87f7202aaa9ec1f96c899c9cf8757b7dc75e86327887\": container with ID starting with 778cf2ee3aadfa4a437e87f7202aaa9ec1f96c899c9cf8757b7dc75e86327887 not found: ID does not exist"
Jul 06 21:59:34.162274 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:34.162239    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a84965d-efea-4843-b5b2-d35f84f66c77-tokenizer-uds\") pod \"2a84965d-efea-4843-b5b2-d35f84f66c77\" (UID: \"2a84965d-efea-4843-b5b2-d35f84f66c77\") "
Jul 06 21:59:34.162453 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:34.162295    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a84965d-efea-4843-b5b2-d35f84f66c77-tls-certs\") pod \"2a84965d-efea-4843-b5b2-d35f84f66c77\" (UID: \"2a84965d-efea-4843-b5b2-d35f84f66c77\") "
Jul 06 21:59:34.162453 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:34.162390    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vz2n\" (UniqueName: \"kubernetes.io/projected/2a84965d-efea-4843-b5b2-d35f84f66c77-kube-api-access-2vz2n\") pod \"2a84965d-efea-4843-b5b2-d35f84f66c77\" (UID: \"2a84965d-efea-4843-b5b2-d35f84f66c77\") "
Jul 06 21:59:34.162571 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:34.162518    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a84965d-efea-4843-b5b2-d35f84f66c77-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "2a84965d-efea-4843-b5b2-d35f84f66c77" (UID: "2a84965d-efea-4843-b5b2-d35f84f66c77"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:59:34.162706 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:34.162686    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2a84965d-efea-4843-b5b2-d35f84f66c77-tokenizer-uds\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:59:34.164332 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:34.164313    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a84965d-efea-4843-b5b2-d35f84f66c77-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a84965d-efea-4843-b5b2-d35f84f66c77" (UID: "2a84965d-efea-4843-b5b2-d35f84f66c77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:59:34.164452 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:34.164431    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a84965d-efea-4843-b5b2-d35f84f66c77-kube-api-access-2vz2n" (OuterVolumeSpecName: "kube-api-access-2vz2n") pod "2a84965d-efea-4843-b5b2-d35f84f66c77" (UID: "2a84965d-efea-4843-b5b2-d35f84f66c77"). InnerVolumeSpecName "kube-api-access-2vz2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:59:34.263593 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:34.263553    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a84965d-efea-4843-b5b2-d35f84f66c77-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:59:34.263593 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:34.263585    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vz2n\" (UniqueName: \"kubernetes.io/projected/2a84965d-efea-4843-b5b2-d35f84f66c77-kube-api-access-2vz2n\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:59:34.428114 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:34.428081    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk"]
Jul 06 21:59:34.432383 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:34.432360    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dnhjpk"]
Jul 06 21:59:35.657553 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:35.657520    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a84965d-efea-4843-b5b2-d35f84f66c77" path="/var/lib/kubelet/pods/2a84965d-efea-4843-b5b2-d35f84f66c77/volumes"
Jul 06 21:59:48.834350 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:48.834302    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr" podUID="c3e5a988-84fe-4482-b0f6-7121b9b349a8" containerName="main" containerID="cri-o://b0c873f0e773ab5bc87fd1d47fb0287bd063bcd5eff5495d72e967dd294287a4" gracePeriod=45
Jul 06 21:59:48.949331 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:48.949292    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr" podUID="c3e5a988-84fe-4482-b0f6-7121b9b349a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 06 21:59:49.084995 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.084922    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:59:49.179685 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.179648    2559 generic.go:356] "Generic (PLEG): container finished" podID="c3e5a988-84fe-4482-b0f6-7121b9b349a8" containerID="b0c873f0e773ab5bc87fd1d47fb0287bd063bcd5eff5495d72e967dd294287a4" exitCode=0
Jul 06 21:59:49.179859 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.179753    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"
Jul 06 21:59:49.179859 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.179765    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr" event={"ID":"c3e5a988-84fe-4482-b0f6-7121b9b349a8","Type":"ContainerDied","Data":"b0c873f0e773ab5bc87fd1d47fb0287bd063bcd5eff5495d72e967dd294287a4"}
Jul 06 21:59:49.179859 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.179808    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr" event={"ID":"c3e5a988-84fe-4482-b0f6-7121b9b349a8","Type":"ContainerDied","Data":"a783d100619946858464d6d1272f259bb1f337af38af8a75e667e89919e90add"}
Jul 06 21:59:49.179859 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.179830    2559 scope.go:117] "RemoveContainer" containerID="b0c873f0e773ab5bc87fd1d47fb0287bd063bcd5eff5495d72e967dd294287a4"
Jul 06 21:59:49.188031 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.188006    2559 scope.go:117] "RemoveContainer" containerID="b0c873f0e773ab5bc87fd1d47fb0287bd063bcd5eff5495d72e967dd294287a4"
Jul 06 21:59:49.188296 ip-10-0-134-202 kubenswrapper[2559]: E0706 21:59:49.188273    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c873f0e773ab5bc87fd1d47fb0287bd063bcd5eff5495d72e967dd294287a4\": container with ID starting with b0c873f0e773ab5bc87fd1d47fb0287bd063bcd5eff5495d72e967dd294287a4 not found: ID does not exist" containerID="b0c873f0e773ab5bc87fd1d47fb0287bd063bcd5eff5495d72e967dd294287a4"
Jul 06 21:59:49.188366 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.188303    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c873f0e773ab5bc87fd1d47fb0287bd063bcd5eff5495d72e967dd294287a4"} err="failed to get container status \"b0c873f0e773ab5bc87fd1d47fb0287bd063bcd5eff5495d72e967dd294287a4\": rpc error: code = NotFound desc = could not find container \"b0c873f0e773ab5bc87fd1d47fb0287bd063bcd5eff5495d72e967dd294287a4\": container with ID starting with b0c873f0e773ab5bc87fd1d47fb0287bd063bcd5eff5495d72e967dd294287a4 not found: ID does not exist"
Jul 06 21:59:49.207689 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.207661    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3e5a988-84fe-4482-b0f6-7121b9b349a8-tls-certs\") pod \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") "
Jul 06 21:59:49.207843 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.207714    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-tmp-dir\") pod \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") "
Jul 06 21:59:49.207843 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.207831    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-model-cache\") pod \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") "
Jul 06 21:59:49.207924 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.207860    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zjqj\" (UniqueName: \"kubernetes.io/projected/c3e5a988-84fe-4482-b0f6-7121b9b349a8-kube-api-access-9zjqj\") pod \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") "
Jul 06 21:59:49.207924 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.207876    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-dshm\") pod \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") "
Jul 06 21:59:49.207924 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.207899    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-home\") pod \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\" (UID: \"c3e5a988-84fe-4482-b0f6-7121b9b349a8\") "
Jul 06 21:59:49.208082 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.207983    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c3e5a988-84fe-4482-b0f6-7121b9b349a8" (UID: "c3e5a988-84fe-4482-b0f6-7121b9b349a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:59:49.208143 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.208082    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3e5a988-84fe-4482-b0f6-7121b9b349a8" (UID: "c3e5a988-84fe-4482-b0f6-7121b9b349a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:59:49.208264 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.208236    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-home" (OuterVolumeSpecName: "home") pod "c3e5a988-84fe-4482-b0f6-7121b9b349a8" (UID: "c3e5a988-84fe-4482-b0f6-7121b9b349a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:59:49.208394 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.208289    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-tmp-dir\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:59:49.208394 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.208304    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-model-cache\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:59:49.209968 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.209938    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-dshm" (OuterVolumeSpecName: "dshm") pod "c3e5a988-84fe-4482-b0f6-7121b9b349a8" (UID: "c3e5a988-84fe-4482-b0f6-7121b9b349a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:59:49.209968 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.209951    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3e5a988-84fe-4482-b0f6-7121b9b349a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3e5a988-84fe-4482-b0f6-7121b9b349a8" (UID: "c3e5a988-84fe-4482-b0f6-7121b9b349a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:59:49.210105 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.210043    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e5a988-84fe-4482-b0f6-7121b9b349a8-kube-api-access-9zjqj" (OuterVolumeSpecName: "kube-api-access-9zjqj") pod "c3e5a988-84fe-4482-b0f6-7121b9b349a8" (UID: "c3e5a988-84fe-4482-b0f6-7121b9b349a8"). InnerVolumeSpecName "kube-api-access-9zjqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:59:49.309227 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.309188    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3e5a988-84fe-4482-b0f6-7121b9b349a8-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:59:49.309227 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.309221    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zjqj\" (UniqueName: \"kubernetes.io/projected/c3e5a988-84fe-4482-b0f6-7121b9b349a8-kube-api-access-9zjqj\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:59:49.309227 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.309231    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-dshm\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:59:49.309227 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.309240    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3e5a988-84fe-4482-b0f6-7121b9b349a8-home\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 21:59:49.500945 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.500914    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"]
Jul 06 21:59:49.506924 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.506898    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-99knr"]
Jul 06 21:59:49.658422 ip-10-0-134-202 kubenswrapper[2559]: I0706 21:59:49.658392    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3e5a988-84fe-4482-b0f6-7121b9b349a8" path="/var/lib/kubelet/pods/c3e5a988-84fe-4482-b0f6-7121b9b349a8/volumes"
Jul 06 22:00:25.001481 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.001445    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"]
Jul 06 22:00:25.001956 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.001874    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a84965d-efea-4843-b5b2-d35f84f66c77" containerName="main"
Jul 06 22:00:25.001956 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.001887    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a84965d-efea-4843-b5b2-d35f84f66c77" containerName="main"
Jul 06 22:00:25.001956 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.001902    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3e5a988-84fe-4482-b0f6-7121b9b349a8" containerName="main"
Jul 06 22:00:25.001956 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.001907    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3e5a988-84fe-4482-b0f6-7121b9b349a8" containerName="main"
Jul 06 22:00:25.002088 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.001986    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3e5a988-84fe-4482-b0f6-7121b9b349a8" containerName="main"
Jul 06 22:00:25.002088 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.001994    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a84965d-efea-4843-b5b2-d35f84f66c77" containerName="main"
Jul 06 22:00:25.006496 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.006468    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:25.008768 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.008747    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 06 22:00:25.014670 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.014648    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"]
Jul 06 22:00:25.124252 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.124212    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:25.124427 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.124274    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:25.124427 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.124337    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:25.124427 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.124376    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/816fe772-df99-4bff-be76-4fd59f8e70c7-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:25.124427 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.124403    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:25.124572 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.124435    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2lf9\" (UniqueName: \"kubernetes.io/projected/816fe772-df99-4bff-be76-4fd59f8e70c7-kube-api-access-h2lf9\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:25.225531 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.225497    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:25.225531 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.225535    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:25.225769 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.225552    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/816fe772-df99-4bff-be76-4fd59f8e70c7-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:25.225769 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.225677    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:25.225769 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.225755    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2lf9\" (UniqueName: \"kubernetes.io/projected/816fe772-df99-4bff-be76-4fd59f8e70c7-kube-api-access-h2lf9\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:25.225925 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.225884    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:25.226011 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.225988    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:25.226178 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.226149    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:25.226342 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.226211    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:25.228132 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.228107    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:25.228891 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.228867    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/816fe772-df99-4bff-be76-4fd59f8e70c7-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:25.234291 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.234261    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2lf9\" (UniqueName: \"kubernetes.io/projected/816fe772-df99-4bff-be76-4fd59f8e70c7-kube-api-access-h2lf9\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:25.319728 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.319693    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:25.462135 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:25.462097    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"]
Jul 06 22:00:25.463244 ip-10-0-134-202 kubenswrapper[2559]: W0706 22:00:25.463206    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816fe772_df99_4bff_be76_4fd59f8e70c7.slice/crio-9caa30a2808ad642c93b26eb0a5426a8c43934d76eeaf6c3ded4a8661329f433 WatchSource:0}: Error finding container 9caa30a2808ad642c93b26eb0a5426a8c43934d76eeaf6c3ded4a8661329f433: Status 404 returned error can't find the container with id 9caa30a2808ad642c93b26eb0a5426a8c43934d76eeaf6c3ded4a8661329f433
Jul 06 22:00:26.357378 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:26.357342    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt" event={"ID":"816fe772-df99-4bff-be76-4fd59f8e70c7","Type":"ContainerStarted","Data":"30448b427af0d46ca804e01e461bae1e70a27cbd6a332f821a5022f6ef2279e7"}
Jul 06 22:00:26.357791 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:26.357394    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt" event={"ID":"816fe772-df99-4bff-be76-4fd59f8e70c7","Type":"ContainerStarted","Data":"9caa30a2808ad642c93b26eb0a5426a8c43934d76eeaf6c3ded4a8661329f433"}
Jul 06 22:00:26.375968 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:26.375891    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt" podStartSLOduration=2.375870997 podStartE2EDuration="2.375870997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:00:26.375479272 +0000 UTC m=+4275.210366656" watchObservedRunningTime="2026-07-06 22:00:26.375870997 +0000 UTC m=+4275.210758363"
Jul 06 22:00:27.320309 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:27.320276    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:35.320682 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:35.320646    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:35.333278 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:35.333247    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:00:35.411663 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:00:35.411634    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:01:15.820909 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:15.820875    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"]
Jul 06 22:01:30.841883 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:30.841832    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt" podUID="816fe772-df99-4bff-be76-4fd59f8e70c7" containerName="main" containerID="cri-o://30448b427af0d46ca804e01e461bae1e70a27cbd6a332f821a5022f6ef2279e7" gracePeriod=45
Jul 06 22:01:31.107386 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.107359    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:01:31.226652 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.226625    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-tmp-dir\") pod \"816fe772-df99-4bff-be76-4fd59f8e70c7\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") "
Jul 06 22:01:31.226652 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.226656    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-dshm\") pod \"816fe772-df99-4bff-be76-4fd59f8e70c7\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") "
Jul 06 22:01:31.226885 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.226679    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-model-cache\") pod \"816fe772-df99-4bff-be76-4fd59f8e70c7\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") "
Jul 06 22:01:31.226885 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.226721    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2lf9\" (UniqueName: \"kubernetes.io/projected/816fe772-df99-4bff-be76-4fd59f8e70c7-kube-api-access-h2lf9\") pod \"816fe772-df99-4bff-be76-4fd59f8e70c7\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") "
Jul 06 22:01:31.226885 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.226745    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/816fe772-df99-4bff-be76-4fd59f8e70c7-tls-certs\") pod \"816fe772-df99-4bff-be76-4fd59f8e70c7\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") "
Jul 06 22:01:31.226885 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.226768    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-home\") pod \"816fe772-df99-4bff-be76-4fd59f8e70c7\" (UID: \"816fe772-df99-4bff-be76-4fd59f8e70c7\") "
Jul 06 22:01:31.226885 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.226843    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "816fe772-df99-4bff-be76-4fd59f8e70c7" (UID: "816fe772-df99-4bff-be76-4fd59f8e70c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:01:31.227148 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.226986    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-model-cache" (OuterVolumeSpecName: "model-cache") pod "816fe772-df99-4bff-be76-4fd59f8e70c7" (UID: "816fe772-df99-4bff-be76-4fd59f8e70c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:01:31.227148 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.227094    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-tmp-dir\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:01:31.227148 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.227112    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-model-cache\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:01:31.227148 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.227105    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-home" (OuterVolumeSpecName: "home") pod "816fe772-df99-4bff-be76-4fd59f8e70c7" (UID: "816fe772-df99-4bff-be76-4fd59f8e70c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:01:31.228844 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.228811    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/816fe772-df99-4bff-be76-4fd59f8e70c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "816fe772-df99-4bff-be76-4fd59f8e70c7" (UID: "816fe772-df99-4bff-be76-4fd59f8e70c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 22:01:31.229213 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.229190    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-dshm" (OuterVolumeSpecName: "dshm") pod "816fe772-df99-4bff-be76-4fd59f8e70c7" (UID: "816fe772-df99-4bff-be76-4fd59f8e70c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:01:31.229213 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.229195    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816fe772-df99-4bff-be76-4fd59f8e70c7-kube-api-access-h2lf9" (OuterVolumeSpecName: "kube-api-access-h2lf9") pod "816fe772-df99-4bff-be76-4fd59f8e70c7" (UID: "816fe772-df99-4bff-be76-4fd59f8e70c7"). InnerVolumeSpecName "kube-api-access-h2lf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:01:31.327782 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.327740    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2lf9\" (UniqueName: \"kubernetes.io/projected/816fe772-df99-4bff-be76-4fd59f8e70c7-kube-api-access-h2lf9\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:01:31.327782 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.327781    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/816fe772-df99-4bff-be76-4fd59f8e70c7-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:01:31.327782 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.327791    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-home\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:01:31.327782 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.327800    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/816fe772-df99-4bff-be76-4fd59f8e70c7-dshm\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:01:31.674766 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.674736    2559 generic.go:356] "Generic (PLEG): container finished" podID="816fe772-df99-4bff-be76-4fd59f8e70c7" containerID="30448b427af0d46ca804e01e461bae1e70a27cbd6a332f821a5022f6ef2279e7" exitCode=0
Jul 06 22:01:31.674888 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.674778    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"
Jul 06 22:01:31.674888 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.674813    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt" event={"ID":"816fe772-df99-4bff-be76-4fd59f8e70c7","Type":"ContainerDied","Data":"30448b427af0d46ca804e01e461bae1e70a27cbd6a332f821a5022f6ef2279e7"}
Jul 06 22:01:31.674888 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.674862    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt" event={"ID":"816fe772-df99-4bff-be76-4fd59f8e70c7","Type":"ContainerDied","Data":"9caa30a2808ad642c93b26eb0a5426a8c43934d76eeaf6c3ded4a8661329f433"}
Jul 06 22:01:31.674888 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.674884    2559 scope.go:117] "RemoveContainer" containerID="30448b427af0d46ca804e01e461bae1e70a27cbd6a332f821a5022f6ef2279e7"
Jul 06 22:01:31.683428 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.683410    2559 scope.go:117] "RemoveContainer" containerID="30448b427af0d46ca804e01e461bae1e70a27cbd6a332f821a5022f6ef2279e7"
Jul 06 22:01:31.683801 ip-10-0-134-202 kubenswrapper[2559]: E0706 22:01:31.683778    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30448b427af0d46ca804e01e461bae1e70a27cbd6a332f821a5022f6ef2279e7\": container with ID starting with 30448b427af0d46ca804e01e461bae1e70a27cbd6a332f821a5022f6ef2279e7 not found: ID does not exist" containerID="30448b427af0d46ca804e01e461bae1e70a27cbd6a332f821a5022f6ef2279e7"
Jul 06 22:01:31.683902 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.683807    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30448b427af0d46ca804e01e461bae1e70a27cbd6a332f821a5022f6ef2279e7"} err="failed to get container status \"30448b427af0d46ca804e01e461bae1e70a27cbd6a332f821a5022f6ef2279e7\": rpc error: code = NotFound desc = could not find container \"30448b427af0d46ca804e01e461bae1e70a27cbd6a332f821a5022f6ef2279e7\": container with ID starting with 30448b427af0d46ca804e01e461bae1e70a27cbd6a332f821a5022f6ef2279e7 not found: ID does not exist"
Jul 06 22:01:31.694389 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.694372    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"]
Jul 06 22:01:31.702631 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:31.700137    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-5b4tt"]
Jul 06 22:01:33.658440 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:01:33.658407    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="816fe772-df99-4bff-be76-4fd59f8e70c7" path="/var/lib/kubelet/pods/816fe772-df99-4bff-be76-4fd59f8e70c7/volumes"
Jul 06 22:02:14.629449 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.629416    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"]
Jul 06 22:02:14.630021 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.629789    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="816fe772-df99-4bff-be76-4fd59f8e70c7" containerName="main"
Jul 06 22:02:14.630021 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.629800    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="816fe772-df99-4bff-be76-4fd59f8e70c7" containerName="main"
Jul 06 22:02:14.630021 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.629876    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="816fe772-df99-4bff-be76-4fd59f8e70c7" containerName="main"
Jul 06 22:02:14.634229 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.634208    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:14.636562 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.636542    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-ch9br\""
Jul 06 22:02:14.636699 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.636684    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 06 22:02:14.648169 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.648065    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"]
Jul 06 22:02:14.732152 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.732117    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:14.732152 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.732156    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt825\" (UniqueName: \"kubernetes.io/projected/2e611b2d-3fd3-4924-bb84-f6f0b228d534-kube-api-access-qt825\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:14.732352 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.732205    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e611b2d-3fd3-4924-bb84-f6f0b228d534-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:14.732352 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.732266    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:14.732352 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.732301    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:14.732451 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.732356    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:14.832862 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.832825    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:14.833044 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.832890    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:14.833044 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.832917    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:14.833044 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.832939    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt825\" (UniqueName: \"kubernetes.io/projected/2e611b2d-3fd3-4924-bb84-f6f0b228d534-kube-api-access-qt825\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:14.833044 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.832971    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e611b2d-3fd3-4924-bb84-f6f0b228d534-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:14.833044 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.832996    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:14.833397 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.833372    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:14.833570 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.833532    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:14.833787 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.833537    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:14.835587 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.835563    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:14.835984 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.835959    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e611b2d-3fd3-4924-bb84-f6f0b228d534-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:14.846760 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.846740    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt825\" (UniqueName: \"kubernetes.io/projected/2e611b2d-3fd3-4924-bb84-f6f0b228d534-kube-api-access-qt825\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:14.855200 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.855179    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8"]
Jul 06 22:02:14.858869 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.858851    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8"
Jul 06 22:02:14.861160 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.861143    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-mskd5\""
Jul 06 22:02:14.868454 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.868435    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8"]
Jul 06 22:02:14.933521 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.933430    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf2a87f7-0a0b-4706-a550-578e1239b679-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8\" (UID: \"cf2a87f7-0a0b-4706-a550-578e1239b679\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8"
Jul 06 22:02:14.933521 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.933495    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cf2a87f7-0a0b-4706-a550-578e1239b679-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8\" (UID: \"cf2a87f7-0a0b-4706-a550-578e1239b679\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8"
Jul 06 22:02:14.933747 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.933564    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfbdp\" (UniqueName: \"kubernetes.io/projected/cf2a87f7-0a0b-4706-a550-578e1239b679-kube-api-access-kfbdp\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8\" (UID: \"cf2a87f7-0a0b-4706-a550-578e1239b679\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8"
Jul 06 22:02:14.943648 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:14.943603    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:15.034269 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:15.034225    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf2a87f7-0a0b-4706-a550-578e1239b679-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8\" (UID: \"cf2a87f7-0a0b-4706-a550-578e1239b679\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8"
Jul 06 22:02:15.034410 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:15.034316    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cf2a87f7-0a0b-4706-a550-578e1239b679-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8\" (UID: \"cf2a87f7-0a0b-4706-a550-578e1239b679\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8"
Jul 06 22:02:15.034410 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:15.034371    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfbdp\" (UniqueName: \"kubernetes.io/projected/cf2a87f7-0a0b-4706-a550-578e1239b679-kube-api-access-kfbdp\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8\" (UID: \"cf2a87f7-0a0b-4706-a550-578e1239b679\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8"
Jul 06 22:02:15.034872 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:15.034831    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cf2a87f7-0a0b-4706-a550-578e1239b679-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8\" (UID: \"cf2a87f7-0a0b-4706-a550-578e1239b679\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8"
Jul 06 22:02:15.037124 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:15.037097    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf2a87f7-0a0b-4706-a550-578e1239b679-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8\" (UID: \"cf2a87f7-0a0b-4706-a550-578e1239b679\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8"
Jul 06 22:02:15.043907 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:15.043786    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfbdp\" (UniqueName: \"kubernetes.io/projected/cf2a87f7-0a0b-4706-a550-578e1239b679-kube-api-access-kfbdp\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8\" (UID: \"cf2a87f7-0a0b-4706-a550-578e1239b679\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8"
Jul 06 22:02:15.085922 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:15.085899    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"]
Jul 06 22:02:15.088391 ip-10-0-134-202 kubenswrapper[2559]: W0706 22:02:15.088357    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e611b2d_3fd3_4924_bb84_f6f0b228d534.slice/crio-93d50bad3055feb726a8c90b3f00918d41e8854228f0aa8ececc0c230a06cdd5 WatchSource:0}: Error finding container 93d50bad3055feb726a8c90b3f00918d41e8854228f0aa8ececc0c230a06cdd5: Status 404 returned error can't find the container with id 93d50bad3055feb726a8c90b3f00918d41e8854228f0aa8ececc0c230a06cdd5
Jul 06 22:02:15.090482 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:15.090468    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 22:02:15.169230 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:15.169203    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8"
Jul 06 22:02:15.302396 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:15.302372    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8"]
Jul 06 22:02:15.304950 ip-10-0-134-202 kubenswrapper[2559]: W0706 22:02:15.304924    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf2a87f7_0a0b_4706_a550_578e1239b679.slice/crio-f4ec777fa20d1db6ebe1336cb3695545896094c9aaba795fc4fde58e7c6a5a5c WatchSource:0}: Error finding container f4ec777fa20d1db6ebe1336cb3695545896094c9aaba795fc4fde58e7c6a5a5c: Status 404 returned error can't find the container with id f4ec777fa20d1db6ebe1336cb3695545896094c9aaba795fc4fde58e7c6a5a5c
Jul 06 22:02:15.882976 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:15.882939    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5" event={"ID":"2e611b2d-3fd3-4924-bb84-f6f0b228d534","Type":"ContainerStarted","Data":"93d50bad3055feb726a8c90b3f00918d41e8854228f0aa8ececc0c230a06cdd5"}
Jul 06 22:02:15.884467 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:15.884442    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8" event={"ID":"cf2a87f7-0a0b-4706-a550-578e1239b679","Type":"ContainerStarted","Data":"059c9ae78e90fa6596d7e5a2497ac8ded03a1efe2fc1432b5a68c9bd46d4c764"}
Jul 06 22:02:15.884582 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:15.884484    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8" event={"ID":"cf2a87f7-0a0b-4706-a550-578e1239b679","Type":"ContainerStarted","Data":"f4ec777fa20d1db6ebe1336cb3695545896094c9aaba795fc4fde58e7c6a5a5c"}
Jul 06 22:02:15.884582 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:15.884531    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8"
Jul 06 22:02:15.901794 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:15.901754    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8" podStartSLOduration=1.901738381 podStartE2EDuration="1.901738381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:02:15.900752193 +0000 UTC m=+4384.735639570" watchObservedRunningTime="2026-07-06 22:02:15.901738381 +0000 UTC m=+4384.736625763"
Jul 06 22:02:16.890107 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:16.890061    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5" event={"ID":"2e611b2d-3fd3-4924-bb84-f6f0b228d534","Type":"ContainerStarted","Data":"415a34e22fdf020c2d80cd1ec8b909ca77f0cc476dd4af0b8fecb3dbc0fdb852"}
Jul 06 22:02:16.890583 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:16.890207    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:17.896262 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:17.896223    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5" event={"ID":"2e611b2d-3fd3-4924-bb84-f6f0b228d534","Type":"ContainerStarted","Data":"e2c1308c315e621a5f88a37865383decb2f24296f620d1d5943593df97c8f57b"}
Jul 06 22:02:17.918409 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:17.918358    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5" podStartSLOduration=3.104497954 podStartE2EDuration="3.918341138s" totalImagesPullingTime="813.843184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:02:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:02:17.916668455 +0000 UTC m=+4386.751555835" watchObservedRunningTime="2026-07-06 22:02:17.918341138 +0000 UTC m=+4386.753228504"
Jul 06 22:02:17.944734 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:17.944689    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:24.944470 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:24.944441    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:24.956728 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:24.956704    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:25.949620 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:25.949575    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:25.951075 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:25.951054    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:02:46.892071 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:02:46.891993    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8"
Jul 06 22:03:32.286016 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:32.285981    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"]
Jul 06 22:03:32.289584 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:32.289140    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8"]
Jul 06 22:03:32.299091 ip-10-0-134-202 kubenswrapper[2559]: E0706 22:03:32.299053    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 22:03:32.299091 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:03:32.299091 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 22:03:32.299091 ip-10-0-134-202 kubenswrapper[2559]:  > containerID="059c9ae78e90fa6596d7e5a2497ac8ded03a1efe2fc1432b5a68c9bd46d4c764" cmd=["/bin/sleep","15"]
Jul 06 22:03:32.299391 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:32.299113    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8" message="" err=<
Jul 06 22:03:32.299391 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:03:32.299391 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 22:03:32.299391 ip-10-0-134-202 kubenswrapper[2559]:  >
Jul 06 22:03:32.299391 ip-10-0-134-202 kubenswrapper[2559]: E0706 22:03:32.299130    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 22:03:32.299391 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:03:32.299391 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 22:03:32.299391 ip-10-0-134-202 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8" podUID="cf2a87f7-0a0b-4706-a550-578e1239b679" containerName="main" containerID="cri-o://059c9ae78e90fa6596d7e5a2497ac8ded03a1efe2fc1432b5a68c9bd46d4c764"
Jul 06 22:03:32.299391 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:32.299173    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8" podUID="cf2a87f7-0a0b-4706-a550-578e1239b679" containerName="main" containerID="cri-o://059c9ae78e90fa6596d7e5a2497ac8ded03a1efe2fc1432b5a68c9bd46d4c764" gracePeriod=60
Jul 06 22:03:32.553067 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:32.553038    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8"
Jul 06 22:03:32.650640 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:32.650575    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfbdp\" (UniqueName: \"kubernetes.io/projected/cf2a87f7-0a0b-4706-a550-578e1239b679-kube-api-access-kfbdp\") pod \"cf2a87f7-0a0b-4706-a550-578e1239b679\" (UID: \"cf2a87f7-0a0b-4706-a550-578e1239b679\") "
Jul 06 22:03:32.650823 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:32.650676    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cf2a87f7-0a0b-4706-a550-578e1239b679-tokenizer-uds\") pod \"cf2a87f7-0a0b-4706-a550-578e1239b679\" (UID: \"cf2a87f7-0a0b-4706-a550-578e1239b679\") "
Jul 06 22:03:32.650823 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:32.650736    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf2a87f7-0a0b-4706-a550-578e1239b679-tls-certs\") pod \"cf2a87f7-0a0b-4706-a550-578e1239b679\" (UID: \"cf2a87f7-0a0b-4706-a550-578e1239b679\") "
Jul 06 22:03:32.650962 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:32.650933    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2a87f7-0a0b-4706-a550-578e1239b679-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "cf2a87f7-0a0b-4706-a550-578e1239b679" (UID: "cf2a87f7-0a0b-4706-a550-578e1239b679"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:03:32.652863 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:32.652832    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf2a87f7-0a0b-4706-a550-578e1239b679-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cf2a87f7-0a0b-4706-a550-578e1239b679" (UID: "cf2a87f7-0a0b-4706-a550-578e1239b679"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 22:03:32.652863 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:32.652849    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2a87f7-0a0b-4706-a550-578e1239b679-kube-api-access-kfbdp" (OuterVolumeSpecName: "kube-api-access-kfbdp") pod "cf2a87f7-0a0b-4706-a550-578e1239b679" (UID: "cf2a87f7-0a0b-4706-a550-578e1239b679"). InnerVolumeSpecName "kube-api-access-kfbdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:03:32.752665 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:32.752634    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfbdp\" (UniqueName: \"kubernetes.io/projected/cf2a87f7-0a0b-4706-a550-578e1239b679-kube-api-access-kfbdp\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:03:32.752665 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:32.752663    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cf2a87f7-0a0b-4706-a550-578e1239b679-tokenizer-uds\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:03:32.752665 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:32.752673    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf2a87f7-0a0b-4706-a550-578e1239b679-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:03:33.273763 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:33.273727    2559 generic.go:356] "Generic (PLEG): container finished" podID="cf2a87f7-0a0b-4706-a550-578e1239b679" containerID="059c9ae78e90fa6596d7e5a2497ac8ded03a1efe2fc1432b5a68c9bd46d4c764" exitCode=0
Jul 06 22:03:33.273937 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:33.273769    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8"
Jul 06 22:03:33.273937 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:33.273799    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8" event={"ID":"cf2a87f7-0a0b-4706-a550-578e1239b679","Type":"ContainerDied","Data":"059c9ae78e90fa6596d7e5a2497ac8ded03a1efe2fc1432b5a68c9bd46d4c764"}
Jul 06 22:03:33.273937 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:33.273843    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8" event={"ID":"cf2a87f7-0a0b-4706-a550-578e1239b679","Type":"ContainerDied","Data":"f4ec777fa20d1db6ebe1336cb3695545896094c9aaba795fc4fde58e7c6a5a5c"}
Jul 06 22:03:33.273937 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:33.273865    2559 scope.go:117] "RemoveContainer" containerID="059c9ae78e90fa6596d7e5a2497ac8ded03a1efe2fc1432b5a68c9bd46d4c764"
Jul 06 22:03:33.282721 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:33.282696    2559 scope.go:117] "RemoveContainer" containerID="059c9ae78e90fa6596d7e5a2497ac8ded03a1efe2fc1432b5a68c9bd46d4c764"
Jul 06 22:03:33.282992 ip-10-0-134-202 kubenswrapper[2559]: E0706 22:03:33.282972    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"059c9ae78e90fa6596d7e5a2497ac8ded03a1efe2fc1432b5a68c9bd46d4c764\": container with ID starting with 059c9ae78e90fa6596d7e5a2497ac8ded03a1efe2fc1432b5a68c9bd46d4c764 not found: ID does not exist" containerID="059c9ae78e90fa6596d7e5a2497ac8ded03a1efe2fc1432b5a68c9bd46d4c764"
Jul 06 22:03:33.283045 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:33.283002    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"059c9ae78e90fa6596d7e5a2497ac8ded03a1efe2fc1432b5a68c9bd46d4c764"} err="failed to get container status \"059c9ae78e90fa6596d7e5a2497ac8ded03a1efe2fc1432b5a68c9bd46d4c764\": rpc error: code = NotFound desc = could not find container \"059c9ae78e90fa6596d7e5a2497ac8ded03a1efe2fc1432b5a68c9bd46d4c764\": container with ID starting with 059c9ae78e90fa6596d7e5a2497ac8ded03a1efe2fc1432b5a68c9bd46d4c764 not found: ID does not exist"
Jul 06 22:03:33.294482 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:33.294460    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8"]
Jul 06 22:03:33.296197 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:33.296179    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59qr7w8"]
Jul 06 22:03:33.659437 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:33.659396    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf2a87f7-0a0b-4706-a550-578e1239b679" path="/var/lib/kubelet/pods/cf2a87f7-0a0b-4706-a550-578e1239b679/volumes"
Jul 06 22:03:47.308358 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:47.308313    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5" podUID="2e611b2d-3fd3-4924-bb84-f6f0b228d534" containerName="main" containerID="cri-o://e2c1308c315e621a5f88a37865383decb2f24296f620d1d5943593df97c8f57b" gracePeriod=45
Jul 06 22:03:47.426042 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:47.426008    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5" podUID="2e611b2d-3fd3-4924-bb84-f6f0b228d534" containerName="llm-d-routing-sidecar" containerID="cri-o://415a34e22fdf020c2d80cd1ec8b909ca77f0cc476dd4af0b8fecb3dbc0fdb852" gracePeriod=45
Jul 06 22:03:47.691070 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:47.691039    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:03:47.779443 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:47.779406    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-home\") pod \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") "
Jul 06 22:03:47.779652 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:47.779469    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qt825\" (UniqueName: \"kubernetes.io/projected/2e611b2d-3fd3-4924-bb84-f6f0b228d534-kube-api-access-qt825\") pod \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") "
Jul 06 22:03:47.779652 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:47.779544    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e611b2d-3fd3-4924-bb84-f6f0b228d534-tls-certs\") pod \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") "
Jul 06 22:03:47.779652 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:47.779567    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-dshm\") pod \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") "
Jul 06 22:03:47.779652 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:47.779587    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-model-cache\") pod \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") "
Jul 06 22:03:47.779906 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:47.779760    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-home" (OuterVolumeSpecName: "home") pod "2e611b2d-3fd3-4924-bb84-f6f0b228d534" (UID: "2e611b2d-3fd3-4924-bb84-f6f0b228d534"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:03:47.779906 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:47.779818    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-tmp-dir\") pod \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\" (UID: \"2e611b2d-3fd3-4924-bb84-f6f0b228d534\") "
Jul 06 22:03:47.780018 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:47.779942    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e611b2d-3fd3-4924-bb84-f6f0b228d534" (UID: "2e611b2d-3fd3-4924-bb84-f6f0b228d534"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:03:47.780018 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:47.779988    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e611b2d-3fd3-4924-bb84-f6f0b228d534" (UID: "2e611b2d-3fd3-4924-bb84-f6f0b228d534"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:03:47.780167 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:47.780152    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-tmp-dir\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:03:47.780214 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:47.780169    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-home\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:03:47.780214 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:47.780178    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-model-cache\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:03:47.781736 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:47.781716    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e611b2d-3fd3-4924-bb84-f6f0b228d534-kube-api-access-qt825" (OuterVolumeSpecName: "kube-api-access-qt825") pod "2e611b2d-3fd3-4924-bb84-f6f0b228d534" (UID: "2e611b2d-3fd3-4924-bb84-f6f0b228d534"). InnerVolumeSpecName "kube-api-access-qt825". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:03:47.782134 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:47.782108    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e611b2d-3fd3-4924-bb84-f6f0b228d534-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e611b2d-3fd3-4924-bb84-f6f0b228d534" (UID: "2e611b2d-3fd3-4924-bb84-f6f0b228d534"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 22:03:47.782134 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:47.782112    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-dshm" (OuterVolumeSpecName: "dshm") pod "2e611b2d-3fd3-4924-bb84-f6f0b228d534" (UID: "2e611b2d-3fd3-4924-bb84-f6f0b228d534"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:03:47.880810 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:47.880721    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt825\" (UniqueName: \"kubernetes.io/projected/2e611b2d-3fd3-4924-bb84-f6f0b228d534-kube-api-access-qt825\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:03:47.880810 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:47.880751    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e611b2d-3fd3-4924-bb84-f6f0b228d534-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:03:47.880810 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:47.880761    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e611b2d-3fd3-4924-bb84-f6f0b228d534-dshm\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:03:48.349523 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:48.349487    2559 generic.go:356] "Generic (PLEG): container finished" podID="2e611b2d-3fd3-4924-bb84-f6f0b228d534" containerID="e2c1308c315e621a5f88a37865383decb2f24296f620d1d5943593df97c8f57b" exitCode=0
Jul 06 22:03:48.349523 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:48.349517    2559 generic.go:356] "Generic (PLEG): container finished" podID="2e611b2d-3fd3-4924-bb84-f6f0b228d534" containerID="415a34e22fdf020c2d80cd1ec8b909ca77f0cc476dd4af0b8fecb3dbc0fdb852" exitCode=0
Jul 06 22:03:48.350053 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:48.349536    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"
Jul 06 22:03:48.350053 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:48.349582    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5" event={"ID":"2e611b2d-3fd3-4924-bb84-f6f0b228d534","Type":"ContainerDied","Data":"e2c1308c315e621a5f88a37865383decb2f24296f620d1d5943593df97c8f57b"}
Jul 06 22:03:48.350053 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:48.349658    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5" event={"ID":"2e611b2d-3fd3-4924-bb84-f6f0b228d534","Type":"ContainerDied","Data":"415a34e22fdf020c2d80cd1ec8b909ca77f0cc476dd4af0b8fecb3dbc0fdb852"}
Jul 06 22:03:48.350053 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:48.349678    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5" event={"ID":"2e611b2d-3fd3-4924-bb84-f6f0b228d534","Type":"ContainerDied","Data":"93d50bad3055feb726a8c90b3f00918d41e8854228f0aa8ececc0c230a06cdd5"}
Jul 06 22:03:48.350053 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:48.349702    2559 scope.go:117] "RemoveContainer" containerID="e2c1308c315e621a5f88a37865383decb2f24296f620d1d5943593df97c8f57b"
Jul 06 22:03:48.359094 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:48.359072    2559 scope.go:117] "RemoveContainer" containerID="415a34e22fdf020c2d80cd1ec8b909ca77f0cc476dd4af0b8fecb3dbc0fdb852"
Jul 06 22:03:48.367217 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:48.367196    2559 scope.go:117] "RemoveContainer" containerID="e2c1308c315e621a5f88a37865383decb2f24296f620d1d5943593df97c8f57b"
Jul 06 22:03:48.367491 ip-10-0-134-202 kubenswrapper[2559]: E0706 22:03:48.367471    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2c1308c315e621a5f88a37865383decb2f24296f620d1d5943593df97c8f57b\": container with ID starting with e2c1308c315e621a5f88a37865383decb2f24296f620d1d5943593df97c8f57b not found: ID does not exist" containerID="e2c1308c315e621a5f88a37865383decb2f24296f620d1d5943593df97c8f57b"
Jul 06 22:03:48.367562 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:48.367500    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c1308c315e621a5f88a37865383decb2f24296f620d1d5943593df97c8f57b"} err="failed to get container status \"e2c1308c315e621a5f88a37865383decb2f24296f620d1d5943593df97c8f57b\": rpc error: code = NotFound desc = could not find container \"e2c1308c315e621a5f88a37865383decb2f24296f620d1d5943593df97c8f57b\": container with ID starting with e2c1308c315e621a5f88a37865383decb2f24296f620d1d5943593df97c8f57b not found: ID does not exist"
Jul 06 22:03:48.367562 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:48.367518    2559 scope.go:117] "RemoveContainer" containerID="415a34e22fdf020c2d80cd1ec8b909ca77f0cc476dd4af0b8fecb3dbc0fdb852"
Jul 06 22:03:48.367800 ip-10-0-134-202 kubenswrapper[2559]: E0706 22:03:48.367782    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415a34e22fdf020c2d80cd1ec8b909ca77f0cc476dd4af0b8fecb3dbc0fdb852\": container with ID starting with 415a34e22fdf020c2d80cd1ec8b909ca77f0cc476dd4af0b8fecb3dbc0fdb852 not found: ID does not exist" containerID="415a34e22fdf020c2d80cd1ec8b909ca77f0cc476dd4af0b8fecb3dbc0fdb852"
Jul 06 22:03:48.367847 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:48.367808    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415a34e22fdf020c2d80cd1ec8b909ca77f0cc476dd4af0b8fecb3dbc0fdb852"} err="failed to get container status \"415a34e22fdf020c2d80cd1ec8b909ca77f0cc476dd4af0b8fecb3dbc0fdb852\": rpc error: code = NotFound desc = could not find container \"415a34e22fdf020c2d80cd1ec8b909ca77f0cc476dd4af0b8fecb3dbc0fdb852\": container with ID starting with 415a34e22fdf020c2d80cd1ec8b909ca77f0cc476dd4af0b8fecb3dbc0fdb852 not found: ID does not exist"
Jul 06 22:03:48.367847 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:48.367824    2559 scope.go:117] "RemoveContainer" containerID="e2c1308c315e621a5f88a37865383decb2f24296f620d1d5943593df97c8f57b"
Jul 06 22:03:48.368019 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:48.368003    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c1308c315e621a5f88a37865383decb2f24296f620d1d5943593df97c8f57b"} err="failed to get container status \"e2c1308c315e621a5f88a37865383decb2f24296f620d1d5943593df97c8f57b\": rpc error: code = NotFound desc = could not find container \"e2c1308c315e621a5f88a37865383decb2f24296f620d1d5943593df97c8f57b\": container with ID starting with e2c1308c315e621a5f88a37865383decb2f24296f620d1d5943593df97c8f57b not found: ID does not exist"
Jul 06 22:03:48.368066 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:48.368019    2559 scope.go:117] "RemoveContainer" containerID="415a34e22fdf020c2d80cd1ec8b909ca77f0cc476dd4af0b8fecb3dbc0fdb852"
Jul 06 22:03:48.368231 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:48.368216    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415a34e22fdf020c2d80cd1ec8b909ca77f0cc476dd4af0b8fecb3dbc0fdb852"} err="failed to get container status \"415a34e22fdf020c2d80cd1ec8b909ca77f0cc476dd4af0b8fecb3dbc0fdb852\": rpc error: code = NotFound desc = could not find container \"415a34e22fdf020c2d80cd1ec8b909ca77f0cc476dd4af0b8fecb3dbc0fdb852\": container with ID starting with 415a34e22fdf020c2d80cd1ec8b909ca77f0cc476dd4af0b8fecb3dbc0fdb852 not found: ID does not exist"
Jul 06 22:03:48.372969 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:48.372947    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"]
Jul 06 22:03:48.376696 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:48.376674    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-bgqq5"]
Jul 06 22:03:49.658047 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:03:49.658005    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e611b2d-3fd3-4924-bb84-f6f0b228d534" path="/var/lib/kubelet/pods/2e611b2d-3fd3-4924-bb84-f6f0b228d534/volumes"
Jul 06 22:04:08.334716 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.334632    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"]
Jul 06 22:04:08.335251 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.335040    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e611b2d-3fd3-4924-bb84-f6f0b228d534" containerName="main"
Jul 06 22:04:08.335251 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.335056    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e611b2d-3fd3-4924-bb84-f6f0b228d534" containerName="main"
Jul 06 22:04:08.335251 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.335068    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e611b2d-3fd3-4924-bb84-f6f0b228d534" containerName="llm-d-routing-sidecar"
Jul 06 22:04:08.335251 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.335076    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e611b2d-3fd3-4924-bb84-f6f0b228d534" containerName="llm-d-routing-sidecar"
Jul 06 22:04:08.335251 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.335087    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf2a87f7-0a0b-4706-a550-578e1239b679" containerName="main"
Jul 06 22:04:08.335251 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.335096    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf2a87f7-0a0b-4706-a550-578e1239b679" containerName="main"
Jul 06 22:04:08.335251 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.335199    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e611b2d-3fd3-4924-bb84-f6f0b228d534" containerName="main"
Jul 06 22:04:08.335251 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.335211    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf2a87f7-0a0b-4706-a550-578e1239b679" containerName="main"
Jul 06 22:04:08.335251 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.335219    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e611b2d-3fd3-4924-bb84-f6f0b228d534" containerName="llm-d-routing-sidecar"
Jul 06 22:04:08.338427 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.338411    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:08.340782 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.340758    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 06 22:04:08.341505 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.341484    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-29ngl\""
Jul 06 22:04:08.353119 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.353093    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"]
Jul 06 22:04:08.465025 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.464994    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:08.465025 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.465027    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mgfj\" (UniqueName: \"kubernetes.io/projected/25049e0a-534e-4afc-b499-6176ad9bf2b6-kube-api-access-5mgfj\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:08.465261 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.465050    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:08.465261 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.465167    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:08.465261 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.465202    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:08.465261 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.465253    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25049e0a-534e-4afc-b499-6176ad9bf2b6-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:08.566227 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.566177    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:08.566412 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.566245    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:08.566412 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.566294    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25049e0a-534e-4afc-b499-6176ad9bf2b6-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:08.566412 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.566356    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:08.566412 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.566382    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mgfj\" (UniqueName: \"kubernetes.io/projected/25049e0a-534e-4afc-b499-6176ad9bf2b6-kube-api-access-5mgfj\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:08.566412 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.566411    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:08.566712 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.566691    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:08.566800 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.566777    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:08.566857 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.566807    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:08.568666 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.568638    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:08.568869 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.568851    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25049e0a-534e-4afc-b499-6176ad9bf2b6-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:08.575401 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.575381    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mgfj\" (UniqueName: \"kubernetes.io/projected/25049e0a-534e-4afc-b499-6176ad9bf2b6-kube-api-access-5mgfj\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:08.633164 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.633083    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2"]
Jul 06 22:04:08.637162 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.637144    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2"
Jul 06 22:04:08.639687 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.639662    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-7pvlp\""
Jul 06 22:04:08.645462 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.645439    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2"]
Jul 06 22:04:08.649604 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.649582    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:08.768814 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.768783    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvrjt\" (UniqueName: \"kubernetes.io/projected/2e47dcdd-678a-4377-9501-9dd80f09469f-kube-api-access-jvrjt\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2\" (UID: \"2e47dcdd-678a-4377-9501-9dd80f09469f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2"
Jul 06 22:04:08.769001 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.768866    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e47dcdd-678a-4377-9501-9dd80f09469f-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2\" (UID: \"2e47dcdd-678a-4377-9501-9dd80f09469f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2"
Jul 06 22:04:08.769001 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.768892    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2e47dcdd-678a-4377-9501-9dd80f09469f-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2\" (UID: \"2e47dcdd-678a-4377-9501-9dd80f09469f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2"
Jul 06 22:04:08.793241 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.793218    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"]
Jul 06 22:04:08.795479 ip-10-0-134-202 kubenswrapper[2559]: W0706 22:04:08.795453    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25049e0a_534e_4afc_b499_6176ad9bf2b6.slice/crio-933b203ff4c35ccfd68f8e7adce1e1e44e4644f93d9b1b88a3d9cd9b702c9972 WatchSource:0}: Error finding container 933b203ff4c35ccfd68f8e7adce1e1e44e4644f93d9b1b88a3d9cd9b702c9972: Status 404 returned error can't find the container with id 933b203ff4c35ccfd68f8e7adce1e1e44e4644f93d9b1b88a3d9cd9b702c9972
Jul 06 22:04:08.869348 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.869310    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvrjt\" (UniqueName: \"kubernetes.io/projected/2e47dcdd-678a-4377-9501-9dd80f09469f-kube-api-access-jvrjt\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2\" (UID: \"2e47dcdd-678a-4377-9501-9dd80f09469f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2"
Jul 06 22:04:08.869544 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.869470    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e47dcdd-678a-4377-9501-9dd80f09469f-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2\" (UID: \"2e47dcdd-678a-4377-9501-9dd80f09469f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2"
Jul 06 22:04:08.869544 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.869500    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2e47dcdd-678a-4377-9501-9dd80f09469f-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2\" (UID: \"2e47dcdd-678a-4377-9501-9dd80f09469f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2"
Jul 06 22:04:08.869926 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.869904    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2e47dcdd-678a-4377-9501-9dd80f09469f-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2\" (UID: \"2e47dcdd-678a-4377-9501-9dd80f09469f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2"
Jul 06 22:04:08.872103 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.872079    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e47dcdd-678a-4377-9501-9dd80f09469f-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2\" (UID: \"2e47dcdd-678a-4377-9501-9dd80f09469f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2"
Jul 06 22:04:08.877574 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.877552    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvrjt\" (UniqueName: \"kubernetes.io/projected/2e47dcdd-678a-4377-9501-9dd80f09469f-kube-api-access-jvrjt\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2\" (UID: \"2e47dcdd-678a-4377-9501-9dd80f09469f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2"
Jul 06 22:04:08.947385 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:08.947280    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2"
Jul 06 22:04:09.087850 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:09.087824    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2"]
Jul 06 22:04:09.090013 ip-10-0-134-202 kubenswrapper[2559]: W0706 22:04:09.089970    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e47dcdd_678a_4377_9501_9dd80f09469f.slice/crio-8c4280d811f76c67abe72f4e1bf803423cca04ab7dbd946e0a817899b1a76b5e WatchSource:0}: Error finding container 8c4280d811f76c67abe72f4e1bf803423cca04ab7dbd946e0a817899b1a76b5e: Status 404 returned error can't find the container with id 8c4280d811f76c67abe72f4e1bf803423cca04ab7dbd946e0a817899b1a76b5e
Jul 06 22:04:09.455154 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:09.455111    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2" event={"ID":"2e47dcdd-678a-4377-9501-9dd80f09469f","Type":"ContainerStarted","Data":"00497257e95cc4dbf2ddf410b5db8779e24cfec8754aaba38f5f256f34f4c642"}
Jul 06 22:04:09.455634 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:09.455168    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2" event={"ID":"2e47dcdd-678a-4377-9501-9dd80f09469f","Type":"ContainerStarted","Data":"8c4280d811f76c67abe72f4e1bf803423cca04ab7dbd946e0a817899b1a76b5e"}
Jul 06 22:04:09.455634 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:09.455221    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2"
Jul 06 22:04:09.456639 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:09.456597    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd" event={"ID":"25049e0a-534e-4afc-b499-6176ad9bf2b6","Type":"ContainerStarted","Data":"fd8115e75607f146808400c4baf96c7aa312c84179b2adf84c0067e542d219f3"}
Jul 06 22:04:09.456733 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:09.456647    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd" event={"ID":"25049e0a-534e-4afc-b499-6176ad9bf2b6","Type":"ContainerStarted","Data":"933b203ff4c35ccfd68f8e7adce1e1e44e4644f93d9b1b88a3d9cd9b702c9972"}
Jul 06 22:04:09.456733 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:09.456667    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:09.478943 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:09.478895    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2" podStartSLOduration=1.478880394 podStartE2EDuration="1.478880394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:04:09.477214855 +0000 UTC m=+4498.312102219" watchObservedRunningTime="2026-07-06 22:04:09.478880394 +0000 UTC m=+4498.313767759"
Jul 06 22:04:10.464277 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:10.464221    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd" event={"ID":"25049e0a-534e-4afc-b499-6176ad9bf2b6","Type":"ContainerStarted","Data":"479033476f78577ca2c302560a349469ba4a2106ac22539a3d0c754e71ceb079"}
Jul 06 22:04:10.487935 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:10.487878    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd" podStartSLOduration=2.487861069 podStartE2EDuration="2.487861069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:04:10.48638137 +0000 UTC m=+4499.321268759" watchObservedRunningTime="2026-07-06 22:04:10.487861069 +0000 UTC m=+4499.322748434"
Jul 06 22:04:10.650588 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:10.650546    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:12.107988 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:12.107955    2559 scope.go:117] "RemoveContainer" containerID="5f57e2aed9c0e9ff2a9beba49656604ac4a6bbc18e5842b69585927cdbd362ab"
Jul 06 22:04:18.650794 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:18.650752    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:18.664114 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:18.664084    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:18.669444 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:18.669412    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:19.518873 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:19.518844    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:04:40.466501 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:40.466469    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2"
Jul 06 22:04:54.411939 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:54.411901    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"]
Jul 06 22:04:54.418463 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:54.418426    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2"]
Jul 06 22:04:54.429126 ip-10-0-134-202 kubenswrapper[2559]: E0706 22:04:54.429082    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 06 22:04:54.429126 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:04:54.429126 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 22:04:54.429126 ip-10-0-134-202 kubenswrapper[2559]:  > containerID="00497257e95cc4dbf2ddf410b5db8779e24cfec8754aaba38f5f256f34f4c642" cmd=["/bin/sleep","15"]
Jul 06 22:04:54.429471 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:54.429148    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2" message="" err=<
Jul 06 22:04:54.429471 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:04:54.429471 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 22:04:54.429471 ip-10-0-134-202 kubenswrapper[2559]:  >
Jul 06 22:04:54.429471 ip-10-0-134-202 kubenswrapper[2559]: E0706 22:04:54.429166    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 06 22:04:54.429471 ip-10-0-134-202 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 06 22:04:54.429471 ip-10-0-134-202 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 06 22:04:54.429471 ip-10-0-134-202 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2" podUID="2e47dcdd-678a-4377-9501-9dd80f09469f" containerName="main" containerID="cri-o://00497257e95cc4dbf2ddf410b5db8779e24cfec8754aaba38f5f256f34f4c642"
Jul 06 22:04:54.429471 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:54.429212    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2" podUID="2e47dcdd-678a-4377-9501-9dd80f09469f" containerName="main" containerID="cri-o://00497257e95cc4dbf2ddf410b5db8779e24cfec8754aaba38f5f256f34f4c642" gracePeriod=60
Jul 06 22:04:54.686776 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:54.686750    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2"
Jul 06 22:04:54.695157 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:54.695132    2559 generic.go:356] "Generic (PLEG): container finished" podID="2e47dcdd-678a-4377-9501-9dd80f09469f" containerID="00497257e95cc4dbf2ddf410b5db8779e24cfec8754aaba38f5f256f34f4c642" exitCode=0
Jul 06 22:04:54.695308 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:54.695168    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2"
Jul 06 22:04:54.695308 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:54.695213    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2" event={"ID":"2e47dcdd-678a-4377-9501-9dd80f09469f","Type":"ContainerDied","Data":"00497257e95cc4dbf2ddf410b5db8779e24cfec8754aaba38f5f256f34f4c642"}
Jul 06 22:04:54.695308 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:54.695268    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2" event={"ID":"2e47dcdd-678a-4377-9501-9dd80f09469f","Type":"ContainerDied","Data":"8c4280d811f76c67abe72f4e1bf803423cca04ab7dbd946e0a817899b1a76b5e"}
Jul 06 22:04:54.695308 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:54.695293    2559 scope.go:117] "RemoveContainer" containerID="00497257e95cc4dbf2ddf410b5db8779e24cfec8754aaba38f5f256f34f4c642"
Jul 06 22:04:54.704482 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:54.704449    2559 scope.go:117] "RemoveContainer" containerID="00497257e95cc4dbf2ddf410b5db8779e24cfec8754aaba38f5f256f34f4c642"
Jul 06 22:04:54.704858 ip-10-0-134-202 kubenswrapper[2559]: E0706 22:04:54.704827    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00497257e95cc4dbf2ddf410b5db8779e24cfec8754aaba38f5f256f34f4c642\": container with ID starting with 00497257e95cc4dbf2ddf410b5db8779e24cfec8754aaba38f5f256f34f4c642 not found: ID does not exist" containerID="00497257e95cc4dbf2ddf410b5db8779e24cfec8754aaba38f5f256f34f4c642"
Jul 06 22:04:54.704971 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:54.704870    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00497257e95cc4dbf2ddf410b5db8779e24cfec8754aaba38f5f256f34f4c642"} err="failed to get container status \"00497257e95cc4dbf2ddf410b5db8779e24cfec8754aaba38f5f256f34f4c642\": rpc error: code = NotFound desc = could not find container \"00497257e95cc4dbf2ddf410b5db8779e24cfec8754aaba38f5f256f34f4c642\": container with ID starting with 00497257e95cc4dbf2ddf410b5db8779e24cfec8754aaba38f5f256f34f4c642 not found: ID does not exist"
Jul 06 22:04:54.802482 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:54.802447    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e47dcdd-678a-4377-9501-9dd80f09469f-tls-certs\") pod \"2e47dcdd-678a-4377-9501-9dd80f09469f\" (UID: \"2e47dcdd-678a-4377-9501-9dd80f09469f\") "
Jul 06 22:04:54.802705 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:54.802516    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2e47dcdd-678a-4377-9501-9dd80f09469f-tokenizer-uds\") pod \"2e47dcdd-678a-4377-9501-9dd80f09469f\" (UID: \"2e47dcdd-678a-4377-9501-9dd80f09469f\") "
Jul 06 22:04:54.802705 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:54.802538    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvrjt\" (UniqueName: \"kubernetes.io/projected/2e47dcdd-678a-4377-9501-9dd80f09469f-kube-api-access-jvrjt\") pod \"2e47dcdd-678a-4377-9501-9dd80f09469f\" (UID: \"2e47dcdd-678a-4377-9501-9dd80f09469f\") "
Jul 06 22:04:54.802879 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:54.802852    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e47dcdd-678a-4377-9501-9dd80f09469f-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "2e47dcdd-678a-4377-9501-9dd80f09469f" (UID: "2e47dcdd-678a-4377-9501-9dd80f09469f"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:04:54.804674 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:54.804649    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e47dcdd-678a-4377-9501-9dd80f09469f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e47dcdd-678a-4377-9501-9dd80f09469f" (UID: "2e47dcdd-678a-4377-9501-9dd80f09469f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 22:04:54.804823 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:54.804804    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e47dcdd-678a-4377-9501-9dd80f09469f-kube-api-access-jvrjt" (OuterVolumeSpecName: "kube-api-access-jvrjt") pod "2e47dcdd-678a-4377-9501-9dd80f09469f" (UID: "2e47dcdd-678a-4377-9501-9dd80f09469f"). InnerVolumeSpecName "kube-api-access-jvrjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:04:54.903463 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:54.903427    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e47dcdd-678a-4377-9501-9dd80f09469f-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:04:54.903463 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:54.903456    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2e47dcdd-678a-4377-9501-9dd80f09469f-tokenizer-uds\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:04:54.903463 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:54.903465    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvrjt\" (UniqueName: \"kubernetes.io/projected/2e47dcdd-678a-4377-9501-9dd80f09469f-kube-api-access-jvrjt\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:04:55.021550 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:55.021512    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2"]
Jul 06 22:04:55.025290 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:55.025264    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedwqxc2"]
Jul 06 22:04:55.658430 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:04:55.658393    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e47dcdd-678a-4377-9501-9dd80f09469f" path="/var/lib/kubelet/pods/2e47dcdd-678a-4377-9501-9dd80f09469f/volumes"
Jul 06 22:05:09.437333 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.437272    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd" podUID="25049e0a-534e-4afc-b499-6176ad9bf2b6" containerName="main" containerID="cri-o://479033476f78577ca2c302560a349469ba4a2106ac22539a3d0c754e71ceb079" gracePeriod=45
Jul 06 22:05:09.508122 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.508077    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd" podUID="25049e0a-534e-4afc-b499-6176ad9bf2b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8001/health\": dial tcp 10.133.0.55:8001: connect: connection refused"
Jul 06 22:05:09.549435 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.549382    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd" podUID="25049e0a-534e-4afc-b499-6176ad9bf2b6" containerName="llm-d-routing-sidecar" containerID="cri-o://fd8115e75607f146808400c4baf96c7aa312c84179b2adf84c0067e542d219f3" gracePeriod=45
Jul 06 22:05:09.770241 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.770209    2559 generic.go:356] "Generic (PLEG): container finished" podID="25049e0a-534e-4afc-b499-6176ad9bf2b6" containerID="479033476f78577ca2c302560a349469ba4a2106ac22539a3d0c754e71ceb079" exitCode=0
Jul 06 22:05:09.770241 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.770237    2559 generic.go:356] "Generic (PLEG): container finished" podID="25049e0a-534e-4afc-b499-6176ad9bf2b6" containerID="fd8115e75607f146808400c4baf96c7aa312c84179b2adf84c0067e542d219f3" exitCode=0
Jul 06 22:05:09.770436 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.770251    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd" event={"ID":"25049e0a-534e-4afc-b499-6176ad9bf2b6","Type":"ContainerDied","Data":"479033476f78577ca2c302560a349469ba4a2106ac22539a3d0c754e71ceb079"}
Jul 06 22:05:09.770436 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.770306    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd" event={"ID":"25049e0a-534e-4afc-b499-6176ad9bf2b6","Type":"ContainerDied","Data":"fd8115e75607f146808400c4baf96c7aa312c84179b2adf84c0067e542d219f3"}
Jul 06 22:05:09.810770 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.810744    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:05:09.945323 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.945292    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25049e0a-534e-4afc-b499-6176ad9bf2b6-tls-certs\") pod \"25049e0a-534e-4afc-b499-6176ad9bf2b6\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") "
Jul 06 22:05:09.945537 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.945345    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-model-cache\") pod \"25049e0a-534e-4afc-b499-6176ad9bf2b6\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") "
Jul 06 22:05:09.945537 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.945369    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-home\") pod \"25049e0a-534e-4afc-b499-6176ad9bf2b6\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") "
Jul 06 22:05:09.945694 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.945551    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-tmp-dir\") pod \"25049e0a-534e-4afc-b499-6176ad9bf2b6\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") "
Jul 06 22:05:09.945694 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.945582    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "25049e0a-534e-4afc-b499-6176ad9bf2b6" (UID: "25049e0a-534e-4afc-b499-6176ad9bf2b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:05:09.945694 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.945597    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-home" (OuterVolumeSpecName: "home") pod "25049e0a-534e-4afc-b499-6176ad9bf2b6" (UID: "25049e0a-534e-4afc-b499-6176ad9bf2b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:05:09.945694 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.945660    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mgfj\" (UniqueName: \"kubernetes.io/projected/25049e0a-534e-4afc-b499-6176ad9bf2b6-kube-api-access-5mgfj\") pod \"25049e0a-534e-4afc-b499-6176ad9bf2b6\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") "
Jul 06 22:05:09.945876 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.945716    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-dshm\") pod \"25049e0a-534e-4afc-b499-6176ad9bf2b6\" (UID: \"25049e0a-534e-4afc-b499-6176ad9bf2b6\") "
Jul 06 22:05:09.945876 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.945799    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "25049e0a-534e-4afc-b499-6176ad9bf2b6" (UID: "25049e0a-534e-4afc-b499-6176ad9bf2b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:05:09.946033 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.946006    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-tmp-dir\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:05:09.946033 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.946035    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-model-cache\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:05:09.946186 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.946048    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-home\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:05:09.947598 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.947572    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25049e0a-534e-4afc-b499-6176ad9bf2b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "25049e0a-534e-4afc-b499-6176ad9bf2b6" (UID: "25049e0a-534e-4afc-b499-6176ad9bf2b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 22:05:09.947796 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.947777    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-dshm" (OuterVolumeSpecName: "dshm") pod "25049e0a-534e-4afc-b499-6176ad9bf2b6" (UID: "25049e0a-534e-4afc-b499-6176ad9bf2b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:05:09.947867 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:09.947777    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25049e0a-534e-4afc-b499-6176ad9bf2b6-kube-api-access-5mgfj" (OuterVolumeSpecName: "kube-api-access-5mgfj") pod "25049e0a-534e-4afc-b499-6176ad9bf2b6" (UID: "25049e0a-534e-4afc-b499-6176ad9bf2b6"). InnerVolumeSpecName "kube-api-access-5mgfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:05:10.046556 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:10.046517    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25049e0a-534e-4afc-b499-6176ad9bf2b6-dshm\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:05:10.046556 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:10.046553    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25049e0a-534e-4afc-b499-6176ad9bf2b6-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:05:10.046795 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:10.046573    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mgfj\" (UniqueName: \"kubernetes.io/projected/25049e0a-534e-4afc-b499-6176ad9bf2b6-kube-api-access-5mgfj\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:05:10.777047 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:10.777006    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd" event={"ID":"25049e0a-534e-4afc-b499-6176ad9bf2b6","Type":"ContainerDied","Data":"933b203ff4c35ccfd68f8e7adce1e1e44e4644f93d9b1b88a3d9cd9b702c9972"}
Jul 06 22:05:10.777047 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:10.777048    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"
Jul 06 22:05:10.777567 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:10.777085    2559 scope.go:117] "RemoveContainer" containerID="479033476f78577ca2c302560a349469ba4a2106ac22539a3d0c754e71ceb079"
Jul 06 22:05:10.788044 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:10.787979    2559 scope.go:117] "RemoveContainer" containerID="fd8115e75607f146808400c4baf96c7aa312c84179b2adf84c0067e542d219f3"
Jul 06 22:05:10.799361 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:10.799335    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"]
Jul 06 22:05:10.802869 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:10.802849    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-zhckd"]
Jul 06 22:05:11.658545 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:11.658510    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25049e0a-534e-4afc-b499-6176ad9bf2b6" path="/var/lib/kubelet/pods/25049e0a-534e-4afc-b499-6176ad9bf2b6/volumes"
Jul 06 22:05:15.479453 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:15.479417    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lbbd8/must-gather-7skvp"]
Jul 06 22:05:15.479864 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:15.479841    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25049e0a-534e-4afc-b499-6176ad9bf2b6" containerName="main"
Jul 06 22:05:15.479864 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:15.479856    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="25049e0a-534e-4afc-b499-6176ad9bf2b6" containerName="main"
Jul 06 22:05:15.479937 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:15.479877    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e47dcdd-678a-4377-9501-9dd80f09469f" containerName="main"
Jul 06 22:05:15.479937 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:15.479883    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e47dcdd-678a-4377-9501-9dd80f09469f" containerName="main"
Jul 06 22:05:15.479937 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:15.479891    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25049e0a-534e-4afc-b499-6176ad9bf2b6" containerName="llm-d-routing-sidecar"
Jul 06 22:05:15.479937 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:15.479896    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="25049e0a-534e-4afc-b499-6176ad9bf2b6" containerName="llm-d-routing-sidecar"
Jul 06 22:05:15.480065 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:15.479974    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="25049e0a-534e-4afc-b499-6176ad9bf2b6" containerName="llm-d-routing-sidecar"
Jul 06 22:05:15.480065 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:15.479984    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="25049e0a-534e-4afc-b499-6176ad9bf2b6" containerName="main"
Jul 06 22:05:15.480065 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:15.479991    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e47dcdd-678a-4377-9501-9dd80f09469f" containerName="main"
Jul 06 22:05:15.484929 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:15.484905    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lbbd8/must-gather-7skvp"
Jul 06 22:05:15.487542 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:15.487516    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lbbd8\"/\"kube-root-ca.crt\""
Jul 06 22:05:15.487542 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:15.487525    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lbbd8\"/\"openshift-service-ca.crt\""
Jul 06 22:05:15.493404 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:15.493377    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lbbd8/must-gather-7skvp"]
Jul 06 22:05:15.598906 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:15.598866    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0bc1a8ec-330f-4fcd-accb-3dae339c5ac9-must-gather-output\") pod \"must-gather-7skvp\" (UID: \"0bc1a8ec-330f-4fcd-accb-3dae339c5ac9\") " pod="openshift-must-gather-lbbd8/must-gather-7skvp"
Jul 06 22:05:15.599099 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:15.598934    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48d9r\" (UniqueName: \"kubernetes.io/projected/0bc1a8ec-330f-4fcd-accb-3dae339c5ac9-kube-api-access-48d9r\") pod \"must-gather-7skvp\" (UID: \"0bc1a8ec-330f-4fcd-accb-3dae339c5ac9\") " pod="openshift-must-gather-lbbd8/must-gather-7skvp"
Jul 06 22:05:15.700283 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:15.700241    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48d9r\" (UniqueName: \"kubernetes.io/projected/0bc1a8ec-330f-4fcd-accb-3dae339c5ac9-kube-api-access-48d9r\") pod \"must-gather-7skvp\" (UID: \"0bc1a8ec-330f-4fcd-accb-3dae339c5ac9\") " pod="openshift-must-gather-lbbd8/must-gather-7skvp"
Jul 06 22:05:15.700497 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:15.700383    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0bc1a8ec-330f-4fcd-accb-3dae339c5ac9-must-gather-output\") pod \"must-gather-7skvp\" (UID: \"0bc1a8ec-330f-4fcd-accb-3dae339c5ac9\") " pod="openshift-must-gather-lbbd8/must-gather-7skvp"
Jul 06 22:05:15.700781 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:15.700759    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0bc1a8ec-330f-4fcd-accb-3dae339c5ac9-must-gather-output\") pod \"must-gather-7skvp\" (UID: \"0bc1a8ec-330f-4fcd-accb-3dae339c5ac9\") " pod="openshift-must-gather-lbbd8/must-gather-7skvp"
Jul 06 22:05:15.708826 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:15.708798    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48d9r\" (UniqueName: \"kubernetes.io/projected/0bc1a8ec-330f-4fcd-accb-3dae339c5ac9-kube-api-access-48d9r\") pod \"must-gather-7skvp\" (UID: \"0bc1a8ec-330f-4fcd-accb-3dae339c5ac9\") " pod="openshift-must-gather-lbbd8/must-gather-7skvp"
Jul 06 22:05:15.794547 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:15.794518    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lbbd8/must-gather-7skvp"
Jul 06 22:05:15.928681 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:15.928654    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lbbd8/must-gather-7skvp"]
Jul 06 22:05:15.931186 ip-10-0-134-202 kubenswrapper[2559]: W0706 22:05:15.931157    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bc1a8ec_330f_4fcd_accb_3dae339c5ac9.slice/crio-50cededf33798e57e72dc5e3703cf51c003faf8c2326985a650de50585eac2e8 WatchSource:0}: Error finding container 50cededf33798e57e72dc5e3703cf51c003faf8c2326985a650de50585eac2e8: Status 404 returned error can't find the container with id 50cededf33798e57e72dc5e3703cf51c003faf8c2326985a650de50585eac2e8
Jul 06 22:05:16.813301 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:16.813230    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lbbd8/must-gather-7skvp" event={"ID":"0bc1a8ec-330f-4fcd-accb-3dae339c5ac9","Type":"ContainerStarted","Data":"50cededf33798e57e72dc5e3703cf51c003faf8c2326985a650de50585eac2e8"}
Jul 06 22:05:22.851114 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:22.851080    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lbbd8/must-gather-7skvp" event={"ID":"0bc1a8ec-330f-4fcd-accb-3dae339c5ac9","Type":"ContainerStarted","Data":"49b2dd67b98947a3692f87c48b9d199fcf264f2f5bcfa8c01ce47c0c5c0c3515"}
Jul 06 22:05:22.851542 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:22.851134    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lbbd8/must-gather-7skvp" event={"ID":"0bc1a8ec-330f-4fcd-accb-3dae339c5ac9","Type":"ContainerStarted","Data":"dc222d3ed8c38ba3ac6f18bdd50c7259d735d95e8ab7beccf42521db91bb69ae"}
Jul 06 22:05:22.869277 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:22.869216    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lbbd8/must-gather-7skvp" podStartSLOduration=1.7081574370000001 podStartE2EDuration="7.869194092s" totalImagesPullingTime="6.161036655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:05:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:05:22.867105028 +0000 UTC m=+4571.701992395" watchObservedRunningTime="2026-07-06 22:05:22.869194092 +0000 UTC m=+4571.704081459"
Jul 06 22:05:31.799576 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:31.799545    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-z62nj_434956f2-26d1-4e99-8e60-363e99f00b0e/storage-initializer/0.log"
Jul 06 22:05:32.669605 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:32.669574    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv_029c2f6a-b869-4b84-b178-3b7dc3eef6fe/main/0.log"
Jul 06 22:05:32.717223 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:32.717160    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-zpwwt_76097b92-270c-426b-b7ec-95a0dfed13e2/istio-proxy/0.log"
Jul 06 22:05:32.734733 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:32.734702    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/1.log"
Jul 06 22:05:32.735100 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:32.735085    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/0.log"
Jul 06 22:05:34.106822 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:34.106782    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-z62nj_434956f2-26d1-4e99-8e60-363e99f00b0e/storage-initializer/0.log"
Jul 06 22:05:34.945220 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:34.945188    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv_029c2f6a-b869-4b84-b178-3b7dc3eef6fe/main/0.log"
Jul 06 22:05:34.994079 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:34.994048    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-zpwwt_76097b92-270c-426b-b7ec-95a0dfed13e2/istio-proxy/0.log"
Jul 06 22:05:35.013788 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:35.013756    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/1.log"
Jul 06 22:05:35.014165 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:35.014148    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/0.log"
Jul 06 22:05:36.445803 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:36.445763    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-z62nj_434956f2-26d1-4e99-8e60-363e99f00b0e/storage-initializer/0.log"
Jul 06 22:05:37.269764 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:37.269732    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv_029c2f6a-b869-4b84-b178-3b7dc3eef6fe/main/0.log"
Jul 06 22:05:37.314860 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:37.314832    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-zpwwt_76097b92-270c-426b-b7ec-95a0dfed13e2/istio-proxy/0.log"
Jul 06 22:05:37.332588 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:37.332562    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/1.log"
Jul 06 22:05:37.332964 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:37.332947    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/0.log"
Jul 06 22:05:38.659298 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:38.659273    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-z62nj_434956f2-26d1-4e99-8e60-363e99f00b0e/storage-initializer/0.log"
Jul 06 22:05:39.497450 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:39.497417    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv_029c2f6a-b869-4b84-b178-3b7dc3eef6fe/main/0.log"
Jul 06 22:05:39.546326 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:39.546298    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-zpwwt_76097b92-270c-426b-b7ec-95a0dfed13e2/istio-proxy/0.log"
Jul 06 22:05:39.562925 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:39.562899    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/1.log"
Jul 06 22:05:39.563269 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:39.563252    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/0.log"
Jul 06 22:05:40.899241 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:40.899216    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-z62nj_434956f2-26d1-4e99-8e60-363e99f00b0e/storage-initializer/0.log"
Jul 06 22:05:41.785477 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:41.785446    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv_029c2f6a-b869-4b84-b178-3b7dc3eef6fe/main/0.log"
Jul 06 22:05:41.850965 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:41.850940    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-zpwwt_76097b92-270c-426b-b7ec-95a0dfed13e2/istio-proxy/0.log"
Jul 06 22:05:41.874173 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:41.874142    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/1.log"
Jul 06 22:05:41.874453 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:41.874438    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/0.log"
Jul 06 22:05:43.315785 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:43.315751    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-z62nj_434956f2-26d1-4e99-8e60-363e99f00b0e/storage-initializer/0.log"
Jul 06 22:05:44.191282 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:44.191255    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv_029c2f6a-b869-4b84-b178-3b7dc3eef6fe/main/0.log"
Jul 06 22:05:44.244387 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:44.244338    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-zpwwt_76097b92-270c-426b-b7ec-95a0dfed13e2/istio-proxy/0.log"
Jul 06 22:05:44.263410 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:44.263374    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/1.log"
Jul 06 22:05:44.263641 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:44.263626    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/0.log"
Jul 06 22:05:45.580310 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:45.580284    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-z62nj_434956f2-26d1-4e99-8e60-363e99f00b0e/storage-initializer/0.log"
Jul 06 22:05:46.407924 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:46.407896    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv_029c2f6a-b869-4b84-b178-3b7dc3eef6fe/main/0.log"
Jul 06 22:05:46.453177 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:46.453140    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-zpwwt_76097b92-270c-426b-b7ec-95a0dfed13e2/istio-proxy/0.log"
Jul 06 22:05:46.470236 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:46.470208    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/1.log"
Jul 06 22:05:46.470469 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:46.470453    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/0.log"
Jul 06 22:05:47.815528 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:47.815501    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-z62nj_434956f2-26d1-4e99-8e60-363e99f00b0e/storage-initializer/0.log"
Jul 06 22:05:48.638560 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:48.638529    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv_029c2f6a-b869-4b84-b178-3b7dc3eef6fe/main/0.log"
Jul 06 22:05:48.686832 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:48.686804    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-zpwwt_76097b92-270c-426b-b7ec-95a0dfed13e2/istio-proxy/0.log"
Jul 06 22:05:48.703684 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:48.703655    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/1.log"
Jul 06 22:05:48.703986 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:48.703968    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/0.log"
Jul 06 22:05:50.024524 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:50.024497    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-z62nj_434956f2-26d1-4e99-8e60-363e99f00b0e/storage-initializer/0.log"
Jul 06 22:05:50.863447 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:50.863418    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv_029c2f6a-b869-4b84-b178-3b7dc3eef6fe/main/0.log"
Jul 06 22:05:50.915782 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:50.915751    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-zpwwt_76097b92-270c-426b-b7ec-95a0dfed13e2/istio-proxy/0.log"
Jul 06 22:05:50.940925 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:50.940887    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/1.log"
Jul 06 22:05:50.941236 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:50.941218    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/0.log"
Jul 06 22:05:52.298246 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:52.298213    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-z62nj_434956f2-26d1-4e99-8e60-363e99f00b0e/storage-initializer/0.log"
Jul 06 22:05:53.133914 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:53.133883    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv_029c2f6a-b869-4b84-b178-3b7dc3eef6fe/main/0.log"
Jul 06 22:05:53.177154 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:53.177125    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-zpwwt_76097b92-270c-426b-b7ec-95a0dfed13e2/istio-proxy/0.log"
Jul 06 22:05:53.193405 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:53.193374    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/1.log"
Jul 06 22:05:53.193828 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:53.193811    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/0.log"
Jul 06 22:05:54.532106 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:54.532078    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-z62nj_434956f2-26d1-4e99-8e60-363e99f00b0e/storage-initializer/0.log"
Jul 06 22:05:55.369626 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:55.369586    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv_029c2f6a-b869-4b84-b178-3b7dc3eef6fe/main/0.log"
Jul 06 22:05:55.417499 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:55.417472    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-zpwwt_76097b92-270c-426b-b7ec-95a0dfed13e2/istio-proxy/0.log"
Jul 06 22:05:55.435731 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:55.435706    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/1.log"
Jul 06 22:05:55.436026 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:55.436014    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/0.log"
Jul 06 22:05:56.789934 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:56.789907    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-z62nj_434956f2-26d1-4e99-8e60-363e99f00b0e/storage-initializer/0.log"
Jul 06 22:05:57.617797 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:57.617765    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv_029c2f6a-b869-4b84-b178-3b7dc3eef6fe/main/0.log"
Jul 06 22:05:57.660672 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:57.660643    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-zpwwt_76097b92-270c-426b-b7ec-95a0dfed13e2/istio-proxy/0.log"
Jul 06 22:05:57.677006 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:57.676976    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/1.log"
Jul 06 22:05:57.677340 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:57.677324    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/0.log"
Jul 06 22:05:59.002472 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:59.002447    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-z62nj_434956f2-26d1-4e99-8e60-363e99f00b0e/storage-initializer/0.log"
Jul 06 22:05:59.829635 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:59.829583    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv_029c2f6a-b869-4b84-b178-3b7dc3eef6fe/main/0.log"
Jul 06 22:05:59.873333 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:59.873302    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-zpwwt_76097b92-270c-426b-b7ec-95a0dfed13e2/istio-proxy/0.log"
Jul 06 22:05:59.889347 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:59.889313    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/1.log"
Jul 06 22:05:59.889645 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:05:59.889625    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/0.log"
Jul 06 22:06:01.257781 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:01.257753    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-z62nj_434956f2-26d1-4e99-8e60-363e99f00b0e/storage-initializer/0.log"
Jul 06 22:06:02.084936 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:02.084909    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv_029c2f6a-b869-4b84-b178-3b7dc3eef6fe/main/0.log"
Jul 06 22:06:02.135824 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:02.135798    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-zpwwt_76097b92-270c-426b-b7ec-95a0dfed13e2/istio-proxy/0.log"
Jul 06 22:06:02.152904 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:02.152873    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/1.log"
Jul 06 22:06:02.153166 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:02.153154    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/0.log"
Jul 06 22:06:03.492591 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:03.492562    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-z62nj_434956f2-26d1-4e99-8e60-363e99f00b0e/storage-initializer/0.log"
Jul 06 22:06:04.352521 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:04.352495    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv_029c2f6a-b869-4b84-b178-3b7dc3eef6fe/main/0.log"
Jul 06 22:06:04.400526 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:04.400493    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-zpwwt_76097b92-270c-426b-b7ec-95a0dfed13e2/istio-proxy/0.log"
Jul 06 22:06:04.421520 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:04.421496    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/1.log"
Jul 06 22:06:04.421950 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:04.421935    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/0.log"
Jul 06 22:06:05.764731 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:05.764700    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-z62nj_434956f2-26d1-4e99-8e60-363e99f00b0e/storage-initializer/0.log"
Jul 06 22:06:06.604475 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:06.604445    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv_029c2f6a-b869-4b84-b178-3b7dc3eef6fe/main/0.log"
Jul 06 22:06:06.658532 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:06.658503    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-zpwwt_76097b92-270c-426b-b7ec-95a0dfed13e2/istio-proxy/0.log"
Jul 06 22:06:06.675313 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:06.675283    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/1.log"
Jul 06 22:06:06.675576 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:06.675559    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-nrf44_047c1603-6763-4cb1-8c0f-8094f55a94b7/istio-proxy/0.log"
Jul 06 22:06:12.645673 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:12.645642    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-nqg8v_dd7b6366-25ec-445a-beed-9ba9205afd53/limitador/0.log"
Jul 06 22:06:12.663414 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:12.663385    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-w2wzk_c2582e4d-80db-4433-9393-eee2761f443c/manager/0.log"
Jul 06 22:06:13.862172 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:13.862140    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-nqg8v_dd7b6366-25ec-445a-beed-9ba9205afd53/limitador/0.log"
Jul 06 22:06:13.879542 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:13.879510    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-w2wzk_c2582e4d-80db-4433-9393-eee2761f443c/manager/0.log"
Jul 06 22:06:15.058059 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:15.058019    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-nqg8v_dd7b6366-25ec-445a-beed-9ba9205afd53/limitador/0.log"
Jul 06 22:06:15.074202 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:15.074166    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-w2wzk_c2582e4d-80db-4433-9393-eee2761f443c/manager/0.log"
Jul 06 22:06:16.255745 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:16.255716    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-nqg8v_dd7b6366-25ec-445a-beed-9ba9205afd53/limitador/0.log"
Jul 06 22:06:16.273274 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:16.273248    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-w2wzk_c2582e4d-80db-4433-9393-eee2761f443c/manager/0.log"
Jul 06 22:06:17.433811 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:17.433783    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-nqg8v_dd7b6366-25ec-445a-beed-9ba9205afd53/limitador/0.log"
Jul 06 22:06:17.450634 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:17.450575    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-w2wzk_c2582e4d-80db-4433-9393-eee2761f443c/manager/0.log"
Jul 06 22:06:18.625956 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:18.625925    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-nqg8v_dd7b6366-25ec-445a-beed-9ba9205afd53/limitador/0.log"
Jul 06 22:06:18.643877 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:18.643852    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-w2wzk_c2582e4d-80db-4433-9393-eee2761f443c/manager/0.log"
Jul 06 22:06:19.806453 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:19.806425    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-nqg8v_dd7b6366-25ec-445a-beed-9ba9205afd53/limitador/0.log"
Jul 06 22:06:19.825631 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:19.825582    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-w2wzk_c2582e4d-80db-4433-9393-eee2761f443c/manager/0.log"
Jul 06 22:06:20.989226 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:20.989193    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-nqg8v_dd7b6366-25ec-445a-beed-9ba9205afd53/limitador/0.log"
Jul 06 22:06:21.006808 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:21.006781    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-w2wzk_c2582e4d-80db-4433-9393-eee2761f443c/manager/0.log"
Jul 06 22:06:22.151482 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:22.151455    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-nqg8v_dd7b6366-25ec-445a-beed-9ba9205afd53/limitador/0.log"
Jul 06 22:06:22.167520 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:22.167483    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-w2wzk_c2582e4d-80db-4433-9393-eee2761f443c/manager/0.log"
Jul 06 22:06:23.304341 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:23.304312    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-nqg8v_dd7b6366-25ec-445a-beed-9ba9205afd53/limitador/0.log"
Jul 06 22:06:23.319932 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:23.319905    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-w2wzk_c2582e4d-80db-4433-9393-eee2761f443c/manager/0.log"
Jul 06 22:06:24.477263 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:24.477234    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-nqg8v_dd7b6366-25ec-445a-beed-9ba9205afd53/limitador/0.log"
Jul 06 22:06:24.496840 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:24.496811    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-w2wzk_c2582e4d-80db-4433-9393-eee2761f443c/manager/0.log"
Jul 06 22:06:25.619742 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:25.619711    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-nqg8v_dd7b6366-25ec-445a-beed-9ba9205afd53/limitador/0.log"
Jul 06 22:06:25.638299 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:25.638266    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-w2wzk_c2582e4d-80db-4433-9393-eee2761f443c/manager/0.log"
Jul 06 22:06:26.859052 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:26.859024    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-nqg8v_dd7b6366-25ec-445a-beed-9ba9205afd53/limitador/0.log"
Jul 06 22:06:26.883977 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:26.883941    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-w2wzk_c2582e4d-80db-4433-9393-eee2761f443c/manager/0.log"
Jul 06 22:06:28.129742 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:28.129708    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-nqg8v_dd7b6366-25ec-445a-beed-9ba9205afd53/limitador/0.log"
Jul 06 22:06:28.156913 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:28.156885    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-w2wzk_c2582e4d-80db-4433-9393-eee2761f443c/manager/0.log"
Jul 06 22:06:29.335076 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:29.335048    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-nqg8v_dd7b6366-25ec-445a-beed-9ba9205afd53/limitador/0.log"
Jul 06 22:06:29.352048 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:29.352013    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-w2wzk_c2582e4d-80db-4433-9393-eee2761f443c/manager/0.log"
Jul 06 22:06:30.492039 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:30.492014    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-nqg8v_dd7b6366-25ec-445a-beed-9ba9205afd53/limitador/0.log"
Jul 06 22:06:30.508232 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:30.508205    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-w2wzk_c2582e4d-80db-4433-9393-eee2761f443c/manager/0.log"
Jul 06 22:06:31.658804 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:31.658775    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-nqg8v_dd7b6366-25ec-445a-beed-9ba9205afd53/limitador/0.log"
Jul 06 22:06:31.675829 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:31.675798    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-w2wzk_c2582e4d-80db-4433-9393-eee2761f443c/manager/0.log"
Jul 06 22:06:32.812409 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:32.812382    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-nqg8v_dd7b6366-25ec-445a-beed-9ba9205afd53/limitador/0.log"
Jul 06 22:06:32.828781 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:32.828750    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-w2wzk_c2582e4d-80db-4433-9393-eee2761f443c/manager/0.log"
Jul 06 22:06:34.017005 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:34.016974    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-nqg8v_dd7b6366-25ec-445a-beed-9ba9205afd53/limitador/0.log"
Jul 06 22:06:34.035090 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:34.035057    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-w2wzk_c2582e4d-80db-4433-9393-eee2761f443c/manager/0.log"
Jul 06 22:06:35.193886 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:35.193856    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-nqg8v_dd7b6366-25ec-445a-beed-9ba9205afd53/limitador/0.log"
Jul 06 22:06:35.210095 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:35.210069    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-w2wzk_c2582e4d-80db-4433-9393-eee2761f443c/manager/0.log"
Jul 06 22:06:36.348677 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:36.348647    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-nqg8v_dd7b6366-25ec-445a-beed-9ba9205afd53/limitador/0.log"
Jul 06 22:06:36.365285 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:36.365254    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-w2wzk_c2582e4d-80db-4433-9393-eee2761f443c/manager/0.log"
Jul 06 22:06:37.253689 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:37.253653    2559 generic.go:356] "Generic (PLEG): container finished" podID="0bc1a8ec-330f-4fcd-accb-3dae339c5ac9" containerID="dc222d3ed8c38ba3ac6f18bdd50c7259d735d95e8ab7beccf42521db91bb69ae" exitCode=0
Jul 06 22:06:37.253894 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:37.253706    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lbbd8/must-gather-7skvp" event={"ID":"0bc1a8ec-330f-4fcd-accb-3dae339c5ac9","Type":"ContainerDied","Data":"dc222d3ed8c38ba3ac6f18bdd50c7259d735d95e8ab7beccf42521db91bb69ae"}
Jul 06 22:06:37.254079 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:37.254066    2559 scope.go:117] "RemoveContainer" containerID="dc222d3ed8c38ba3ac6f18bdd50c7259d735d95e8ab7beccf42521db91bb69ae"
Jul 06 22:06:38.160999 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:38.160968    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-lbbd8_must-gather-7skvp_0bc1a8ec-330f-4fcd-accb-3dae339c5ac9/gather/0.log"
Jul 06 22:06:42.050300 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:42.050266    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-khnhx_c260a02b-d5f8-48c9-894b-44c6310ef087/global-pull-secret-syncer/0.log"
Jul 06 22:06:42.145785 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:42.145742    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2lhx7_c72d664d-d8e7-4a0e-a74d-516f8709e325/konnectivity-agent/0.log"
Jul 06 22:06:42.239701 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:42.239670    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-202.ec2.internal_4b12aea9bf8b99530f8679e8c6d1d16c/haproxy/0.log"
Jul 06 22:06:43.787586 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:43.787547    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-lbbd8/must-gather-7skvp"]
Jul 06 22:06:43.788110 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:43.787800    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-lbbd8/must-gather-7skvp" podUID="0bc1a8ec-330f-4fcd-accb-3dae339c5ac9" containerName="copy" containerID="cri-o://49b2dd67b98947a3692f87c48b9d199fcf264f2f5bcfa8c01ce47c0c5c0c3515" gracePeriod=2
Jul 06 22:06:43.793220 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:43.793195    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-lbbd8/must-gather-7skvp"]
Jul 06 22:06:44.037152 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:44.031825    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-lbbd8_must-gather-7skvp_0bc1a8ec-330f-4fcd-accb-3dae339c5ac9/copy/0.log"
Jul 06 22:06:44.038001 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:44.037815    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lbbd8/must-gather-7skvp"
Jul 06 22:06:44.039903 ip-10-0-134-202 kubenswrapper[2559]: E0706 22:06:44.039871    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-7skvp\" is forbidden: User \"system:node:ip-10-0-134-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-lbbd8\": no relationship found between node 'ip-10-0-134-202.ec2.internal' and this object" podUID="0bc1a8ec-330f-4fcd-accb-3dae339c5ac9" pod="openshift-must-gather-lbbd8/must-gather-7skvp"
Jul 06 22:06:44.048179 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:44.048148    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0bc1a8ec-330f-4fcd-accb-3dae339c5ac9-must-gather-output\") pod \"0bc1a8ec-330f-4fcd-accb-3dae339c5ac9\" (UID: \"0bc1a8ec-330f-4fcd-accb-3dae339c5ac9\") "
Jul 06 22:06:44.048314 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:44.048296    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48d9r\" (UniqueName: \"kubernetes.io/projected/0bc1a8ec-330f-4fcd-accb-3dae339c5ac9-kube-api-access-48d9r\") pod \"0bc1a8ec-330f-4fcd-accb-3dae339c5ac9\" (UID: \"0bc1a8ec-330f-4fcd-accb-3dae339c5ac9\") "
Jul 06 22:06:44.050475 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:44.050448    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc1a8ec-330f-4fcd-accb-3dae339c5ac9-kube-api-access-48d9r" (OuterVolumeSpecName: "kube-api-access-48d9r") pod "0bc1a8ec-330f-4fcd-accb-3dae339c5ac9" (UID: "0bc1a8ec-330f-4fcd-accb-3dae339c5ac9"). InnerVolumeSpecName "kube-api-access-48d9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:06:44.059788 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:44.059765    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc1a8ec-330f-4fcd-accb-3dae339c5ac9-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "0bc1a8ec-330f-4fcd-accb-3dae339c5ac9" (UID: "0bc1a8ec-330f-4fcd-accb-3dae339c5ac9"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:06:44.149596 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:44.149563    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48d9r\" (UniqueName: \"kubernetes.io/projected/0bc1a8ec-330f-4fcd-accb-3dae339c5ac9-kube-api-access-48d9r\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:06:44.149596 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:44.149594    2559 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0bc1a8ec-330f-4fcd-accb-3dae339c5ac9-must-gather-output\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\""
Jul 06 22:06:44.286737 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:44.286708    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-lbbd8_must-gather-7skvp_0bc1a8ec-330f-4fcd-accb-3dae339c5ac9/copy/0.log"
Jul 06 22:06:44.287099 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:44.287075    2559 generic.go:356] "Generic (PLEG): container finished" podID="0bc1a8ec-330f-4fcd-accb-3dae339c5ac9" containerID="49b2dd67b98947a3692f87c48b9d199fcf264f2f5bcfa8c01ce47c0c5c0c3515" exitCode=143
Jul 06 22:06:44.287157 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:44.287123    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lbbd8/must-gather-7skvp"
Jul 06 22:06:44.287195 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:44.287179    2559 scope.go:117] "RemoveContainer" containerID="49b2dd67b98947a3692f87c48b9d199fcf264f2f5bcfa8c01ce47c0c5c0c3515"
Jul 06 22:06:44.289093 ip-10-0-134-202 kubenswrapper[2559]: E0706 22:06:44.289061    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-7skvp\" is forbidden: User \"system:node:ip-10-0-134-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-lbbd8\": no relationship found between node 'ip-10-0-134-202.ec2.internal' and this object" podUID="0bc1a8ec-330f-4fcd-accb-3dae339c5ac9" pod="openshift-must-gather-lbbd8/must-gather-7skvp"
Jul 06 22:06:44.295509 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:44.295492    2559 scope.go:117] "RemoveContainer" containerID="dc222d3ed8c38ba3ac6f18bdd50c7259d735d95e8ab7beccf42521db91bb69ae"
Jul 06 22:06:44.297378 ip-10-0-134-202 kubenswrapper[2559]: E0706 22:06:44.297353    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-7skvp\" is forbidden: User \"system:node:ip-10-0-134-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-lbbd8\": no relationship found between node 'ip-10-0-134-202.ec2.internal' and this object" podUID="0bc1a8ec-330f-4fcd-accb-3dae339c5ac9" pod="openshift-must-gather-lbbd8/must-gather-7skvp"
Jul 06 22:06:44.307702 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:44.307678    2559 scope.go:117] "RemoveContainer" containerID="49b2dd67b98947a3692f87c48b9d199fcf264f2f5bcfa8c01ce47c0c5c0c3515"
Jul 06 22:06:44.307996 ip-10-0-134-202 kubenswrapper[2559]: E0706 22:06:44.307978    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49b2dd67b98947a3692f87c48b9d199fcf264f2f5bcfa8c01ce47c0c5c0c3515\": container with ID starting with 49b2dd67b98947a3692f87c48b9d199fcf264f2f5bcfa8c01ce47c0c5c0c3515 not found: ID does not exist" containerID="49b2dd67b98947a3692f87c48b9d199fcf264f2f5bcfa8c01ce47c0c5c0c3515"
Jul 06 22:06:44.308053 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:44.308005    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49b2dd67b98947a3692f87c48b9d199fcf264f2f5bcfa8c01ce47c0c5c0c3515"} err="failed to get container status \"49b2dd67b98947a3692f87c48b9d199fcf264f2f5bcfa8c01ce47c0c5c0c3515\": rpc error: code = NotFound desc = could not find container \"49b2dd67b98947a3692f87c48b9d199fcf264f2f5bcfa8c01ce47c0c5c0c3515\": container with ID starting with 49b2dd67b98947a3692f87c48b9d199fcf264f2f5bcfa8c01ce47c0c5c0c3515 not found: ID does not exist"
Jul 06 22:06:44.308053 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:44.308025    2559 scope.go:117] "RemoveContainer" containerID="dc222d3ed8c38ba3ac6f18bdd50c7259d735d95e8ab7beccf42521db91bb69ae"
Jul 06 22:06:44.308235 ip-10-0-134-202 kubenswrapper[2559]: E0706 22:06:44.308220    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc222d3ed8c38ba3ac6f18bdd50c7259d735d95e8ab7beccf42521db91bb69ae\": container with ID starting with dc222d3ed8c38ba3ac6f18bdd50c7259d735d95e8ab7beccf42521db91bb69ae not found: ID does not exist" containerID="dc222d3ed8c38ba3ac6f18bdd50c7259d735d95e8ab7beccf42521db91bb69ae"
Jul 06 22:06:44.308274 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:44.308238    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc222d3ed8c38ba3ac6f18bdd50c7259d735d95e8ab7beccf42521db91bb69ae"} err="failed to get container status \"dc222d3ed8c38ba3ac6f18bdd50c7259d735d95e8ab7beccf42521db91bb69ae\": rpc error: code = NotFound desc = could not find container \"dc222d3ed8c38ba3ac6f18bdd50c7259d735d95e8ab7beccf42521db91bb69ae\": container with ID starting with dc222d3ed8c38ba3ac6f18bdd50c7259d735d95e8ab7beccf42521db91bb69ae not found: ID does not exist"
Jul 06 22:06:45.658476 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:45.658434    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bc1a8ec-330f-4fcd-accb-3dae339c5ac9" path="/var/lib/kubelet/pods/0bc1a8ec-330f-4fcd-accb-3dae339c5ac9/volumes"
Jul 06 22:06:47.647108 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:47.647077    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-nqg8v_dd7b6366-25ec-445a-beed-9ba9205afd53/limitador/0.log"
Jul 06 22:06:47.709487 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:47.709456    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-w2wzk_c2582e4d-80db-4433-9393-eee2761f443c/manager/0.log"
Jul 06 22:06:49.104814 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:49.104782    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-s9crg_324b6756-b968-4a97-acda-b44144b81452/cluster-monitoring-operator/0.log"
Jul 06 22:06:49.261132 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:49.261103    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7cc4d4799f-rqsdn_41b6c629-1831-4ee6-a0c6-5fe497398a94/metrics-server/0.log"
Jul 06 22:06:49.439475 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:49.439394    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vzht8_dc4993bc-5e06-4c35-a86a-4e74e46ac3cb/node-exporter/0.log"
Jul 06 22:06:49.466718 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:49.466692    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vzht8_dc4993bc-5e06-4c35-a86a-4e74e46ac3cb/kube-rbac-proxy/0.log"
Jul 06 22:06:49.498173 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:49.498146    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vzht8_dc4993bc-5e06-4c35-a86a-4e74e46ac3cb/init-textfile/0.log"
Jul 06 22:06:49.665954 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:49.665925    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-jfscn_d8afe904-ec15-47fd-abf0-8b6deaa0cf4b/kube-rbac-proxy-main/0.log"
Jul 06 22:06:49.691233 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:49.691130    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-jfscn_d8afe904-ec15-47fd-abf0-8b6deaa0cf4b/kube-rbac-proxy-self/0.log"
Jul 06 22:06:49.714202 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:49.714176    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-jfscn_d8afe904-ec15-47fd-abf0-8b6deaa0cf4b/openshift-state-metrics/0.log"
Jul 06 22:06:49.758326 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:49.758294    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2cce5eb-5268-4340-af2f-5e3e3e383690/prometheus/0.log"
Jul 06 22:06:49.777900 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:49.777870    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2cce5eb-5268-4340-af2f-5e3e3e383690/config-reloader/0.log"
Jul 06 22:06:49.800001 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:49.799977    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2cce5eb-5268-4340-af2f-5e3e3e383690/thanos-sidecar/0.log"
Jul 06 22:06:49.824829 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:49.824804    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2cce5eb-5268-4340-af2f-5e3e3e383690/kube-rbac-proxy-web/0.log"
Jul 06 22:06:49.849585 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:49.849561    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2cce5eb-5268-4340-af2f-5e3e3e383690/kube-rbac-proxy/0.log"
Jul 06 22:06:49.872559 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:49.872528    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2cce5eb-5268-4340-af2f-5e3e3e383690/kube-rbac-proxy-thanos/0.log"
Jul 06 22:06:49.897049 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:49.897024    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d2cce5eb-5268-4340-af2f-5e3e3e383690/init-config-reloader/0.log"
Jul 06 22:06:49.933514 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:49.933465    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-wgb8b_2c05a151-03fa-4c3f-9a81-0c51e9333f54/prometheus-operator/0.log"
Jul 06 22:06:49.958951 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:49.958874    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-wgb8b_2c05a151-03fa-4c3f-9a81-0c51e9333f54/kube-rbac-proxy/0.log"
Jul 06 22:06:49.985776 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:49.985745    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-n7nvx_bb1a3249-22a8-4862-877b-bcaeebf12416/prometheus-operator-admission-webhook/0.log"
Jul 06 22:06:50.098346 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:50.098319    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-747bf868c5-jpjjz_2e837295-2af1-4f36-a83d-eba9074cc1a4/thanos-query/0.log"
Jul 06 22:06:50.122789 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:50.122761    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-747bf868c5-jpjjz_2e837295-2af1-4f36-a83d-eba9074cc1a4/kube-rbac-proxy-web/0.log"
Jul 06 22:06:50.149843 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:50.149816    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-747bf868c5-jpjjz_2e837295-2af1-4f36-a83d-eba9074cc1a4/kube-rbac-proxy/0.log"
Jul 06 22:06:50.172989 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:50.172963    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-747bf868c5-jpjjz_2e837295-2af1-4f36-a83d-eba9074cc1a4/prom-label-proxy/0.log"
Jul 06 22:06:50.199216 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:50.199189    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-747bf868c5-jpjjz_2e837295-2af1-4f36-a83d-eba9074cc1a4/kube-rbac-proxy-rules/0.log"
Jul 06 22:06:50.223095 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:50.223017    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-747bf868c5-jpjjz_2e837295-2af1-4f36-a83d-eba9074cc1a4/kube-rbac-proxy-metrics/0.log"
Jul 06 22:06:51.444127 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:51.444090    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-6btsj_61d59548-c626-4260-b1f3-4ecf218a82df/networking-console-plugin/0.log"
Jul 06 22:06:51.952653 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:51.952599    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-cxbrh_adb77bc9-2a7a-443f-a8ca-8b119a477109/console-operator/0.log"
Jul 06 22:06:52.179933 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.179896    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch"]
Jul 06 22:06:52.180335 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.180317    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bc1a8ec-330f-4fcd-accb-3dae339c5ac9" containerName="copy"
Jul 06 22:06:52.180422 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.180337    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bc1a8ec-330f-4fcd-accb-3dae339c5ac9" containerName="copy"
Jul 06 22:06:52.180422 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.180391    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bc1a8ec-330f-4fcd-accb-3dae339c5ac9" containerName="gather"
Jul 06 22:06:52.180422 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.180400    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bc1a8ec-330f-4fcd-accb-3dae339c5ac9" containerName="gather"
Jul 06 22:06:52.180578 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.180505    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bc1a8ec-330f-4fcd-accb-3dae339c5ac9" containerName="copy"
Jul 06 22:06:52.180578 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.180522    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bc1a8ec-330f-4fcd-accb-3dae339c5ac9" containerName="gather"
Jul 06 22:06:52.183919 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.183867    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch"
Jul 06 22:06:52.186476 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.186450    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-95ll7\"/\"openshift-service-ca.crt\""
Jul 06 22:06:52.186606 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.186520    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-95ll7\"/\"kube-root-ca.crt\""
Jul 06 22:06:52.187232 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.187219    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-95ll7\"/\"default-dockercfg-brp8c\""
Jul 06 22:06:52.191851 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.191830    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch"]
Jul 06 22:06:52.220560 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.220468    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aad47073-7730-4d2d-9adf-080ee21c6fe2-lib-modules\") pod \"perf-node-gather-daemonset-c8bch\" (UID: \"aad47073-7730-4d2d-9adf-080ee21c6fe2\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch"
Jul 06 22:06:52.220560 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.220502    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aad47073-7730-4d2d-9adf-080ee21c6fe2-podres\") pod \"perf-node-gather-daemonset-c8bch\" (UID: \"aad47073-7730-4d2d-9adf-080ee21c6fe2\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch"
Jul 06 22:06:52.220560 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.220522    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9ds2\" (UniqueName: \"kubernetes.io/projected/aad47073-7730-4d2d-9adf-080ee21c6fe2-kube-api-access-k9ds2\") pod \"perf-node-gather-daemonset-c8bch\" (UID: \"aad47073-7730-4d2d-9adf-080ee21c6fe2\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch"
Jul 06 22:06:52.220828 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.220669    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aad47073-7730-4d2d-9adf-080ee21c6fe2-proc\") pod \"perf-node-gather-daemonset-c8bch\" (UID: \"aad47073-7730-4d2d-9adf-080ee21c6fe2\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch"
Jul 06 22:06:52.220828 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.220713    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aad47073-7730-4d2d-9adf-080ee21c6fe2-sys\") pod \"perf-node-gather-daemonset-c8bch\" (UID: \"aad47073-7730-4d2d-9adf-080ee21c6fe2\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch"
Jul 06 22:06:52.321225 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.321184    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aad47073-7730-4d2d-9adf-080ee21c6fe2-sys\") pod \"perf-node-gather-daemonset-c8bch\" (UID: \"aad47073-7730-4d2d-9adf-080ee21c6fe2\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch"
Jul 06 22:06:52.321399 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.321317    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aad47073-7730-4d2d-9adf-080ee21c6fe2-sys\") pod \"perf-node-gather-daemonset-c8bch\" (UID: \"aad47073-7730-4d2d-9adf-080ee21c6fe2\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch"
Jul 06 22:06:52.321399 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.321345    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aad47073-7730-4d2d-9adf-080ee21c6fe2-lib-modules\") pod \"perf-node-gather-daemonset-c8bch\" (UID: \"aad47073-7730-4d2d-9adf-080ee21c6fe2\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch"
Jul 06 22:06:52.321399 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.321380    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aad47073-7730-4d2d-9adf-080ee21c6fe2-podres\") pod \"perf-node-gather-daemonset-c8bch\" (UID: \"aad47073-7730-4d2d-9adf-080ee21c6fe2\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch"
Jul 06 22:06:52.321509 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.321405    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9ds2\" (UniqueName: \"kubernetes.io/projected/aad47073-7730-4d2d-9adf-080ee21c6fe2-kube-api-access-k9ds2\") pod \"perf-node-gather-daemonset-c8bch\" (UID: \"aad47073-7730-4d2d-9adf-080ee21c6fe2\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch"
Jul 06 22:06:52.321509 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.321462    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aad47073-7730-4d2d-9adf-080ee21c6fe2-lib-modules\") pod \"perf-node-gather-daemonset-c8bch\" (UID: \"aad47073-7730-4d2d-9adf-080ee21c6fe2\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch"
Jul 06 22:06:52.321575 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.321529    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aad47073-7730-4d2d-9adf-080ee21c6fe2-proc\") pod \"perf-node-gather-daemonset-c8bch\" (UID: \"aad47073-7730-4d2d-9adf-080ee21c6fe2\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch"
Jul 06 22:06:52.321575 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.321536    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aad47073-7730-4d2d-9adf-080ee21c6fe2-podres\") pod \"perf-node-gather-daemonset-c8bch\" (UID: \"aad47073-7730-4d2d-9adf-080ee21c6fe2\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch"
Jul 06 22:06:52.321685 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.321587    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aad47073-7730-4d2d-9adf-080ee21c6fe2-proc\") pod \"perf-node-gather-daemonset-c8bch\" (UID: \"aad47073-7730-4d2d-9adf-080ee21c6fe2\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch"
Jul 06 22:06:52.331189 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.331162    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9ds2\" (UniqueName: \"kubernetes.io/projected/aad47073-7730-4d2d-9adf-080ee21c6fe2-kube-api-access-k9ds2\") pod \"perf-node-gather-daemonset-c8bch\" (UID: \"aad47073-7730-4d2d-9adf-080ee21c6fe2\") " pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch"
Jul 06 22:06:52.494252 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.494168    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch"
Jul 06 22:06:52.495257 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.495239    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-6zzgr_49e9374b-884a-407f-959d-47e2fcc26e12/download-server/0.log"
Jul 06 22:06:52.635161 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:52.635133    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch"]
Jul 06 22:06:52.637071 ip-10-0-134-202 kubenswrapper[2559]: W0706 22:06:52.637046    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaad47073_7730_4d2d_9adf_080ee21c6fe2.slice/crio-70bca35063447bc81470e94fafef45757b4256d16c8ebccd325067e48d5c2cff WatchSource:0}: Error finding container 70bca35063447bc81470e94fafef45757b4256d16c8ebccd325067e48d5c2cff: Status 404 returned error can't find the container with id 70bca35063447bc81470e94fafef45757b4256d16c8ebccd325067e48d5c2cff
Jul 06 22:06:53.109423 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:53.109382    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-8824w_cd8dab4f-de0e-4c53-ba12-86afda4a1016/volume-data-source-validator/0.log"
Jul 06 22:06:53.335413 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:53.335369    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch" event={"ID":"aad47073-7730-4d2d-9adf-080ee21c6fe2","Type":"ContainerStarted","Data":"151cc577ef095e7630021ad6ff70a146e309f9062ddd7d6750d784b205a7793d"}
Jul 06 22:06:53.335600 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:53.335435    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch" event={"ID":"aad47073-7730-4d2d-9adf-080ee21c6fe2","Type":"ContainerStarted","Data":"70bca35063447bc81470e94fafef45757b4256d16c8ebccd325067e48d5c2cff"}
Jul 06 22:06:53.335600 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:53.335481    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch"
Jul 06 22:06:53.353018 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:53.352966    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch" podStartSLOduration=1.352950896 podStartE2EDuration="1.352950896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:06:53.351182054 +0000 UTC m=+4662.186069446" watchObservedRunningTime="2026-07-06 22:06:53.352950896 +0000 UTC m=+4662.187838304"
Jul 06 22:06:54.179857 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:54.179775    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nm7gc_7c6274f2-a3f1-4130-972b-db17b036e580/dns/0.log"
Jul 06 22:06:54.202689 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:54.202601    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nm7gc_7c6274f2-a3f1-4130-972b-db17b036e580/kube-rbac-proxy/0.log"
Jul 06 22:06:54.277603 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:54.277574    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mzmqn_036a3e17-12a4-418f-b550-752a6ff9694e/dns-node-resolver/0.log"
Jul 06 22:06:54.999815 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:54.999783    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rbr6h_5d4b9056-4c7c-43fd-a2b6-6d617f5cf503/node-ca/0.log"
Jul 06 22:06:56.776094 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:56.776061    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dhdnl_6524b812-b7e8-440c-891d-c219ad395486/serve-healthcheck-canary/0.log"
Jul 06 22:06:57.324570 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:57.324537    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-h89pn_d43e04c1-782e-409e-a666-d1e2994fb8a5/insights-operator/0.log"
Jul 06 22:06:57.326073 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:57.326054    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-h89pn_d43e04c1-782e-409e-a666-d1e2994fb8a5/insights-operator/1.log"
Jul 06 22:06:57.348731 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:57.348692    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gfdzd_cc937b92-57db-4b45-9b3b-ad57d91ca45d/kube-rbac-proxy/0.log"
Jul 06 22:06:57.372649 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:57.372604    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gfdzd_cc937b92-57db-4b45-9b3b-ad57d91ca45d/exporter/0.log"
Jul 06 22:06:57.397038 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:57.397004    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gfdzd_cc937b92-57db-4b45-9b3b-ad57d91ca45d/extractor/0.log"
Jul 06 22:06:59.350284 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:06:59.350257    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-95ll7/perf-node-gather-daemonset-c8bch"
Jul 06 22:07:00.317348 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:07:00.317316    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7694784d57-k44g7_9fc927d2-be1f-43ec-9b2b-5ec63c5d0859/manager/0.log"
Jul 06 22:07:08.820652 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:07:08.820601    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-46dgr_ffd4833a-2f6e-4714-872b-bde81197a610/kube-multus/0.log"
Jul 06 22:07:09.207152 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:07:09.207069    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wf6lk_a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2/kube-multus-additional-cni-plugins/0.log"
Jul 06 22:07:09.229718 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:07:09.229694    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wf6lk_a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2/egress-router-binary-copy/0.log"
Jul 06 22:07:09.253854 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:07:09.253830    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wf6lk_a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2/cni-plugins/0.log"
Jul 06 22:07:09.278389 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:07:09.278358    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wf6lk_a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2/bond-cni-plugin/0.log"
Jul 06 22:07:09.301316 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:07:09.301291    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wf6lk_a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2/routeoverride-cni/0.log"
Jul 06 22:07:09.324140 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:07:09.324112    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wf6lk_a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2/whereabouts-cni-bincopy/0.log"
Jul 06 22:07:09.345837 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:07:09.345807    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wf6lk_a2c1cb68-2ad2-4017-ba94-bfc66a42e7b2/whereabouts-cni/0.log"
Jul 06 22:07:09.494819 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:07:09.494736    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ckj9h_ad24b800-9783-47dc-aaa1-4d2bbc01afb8/network-metrics-daemon/0.log"
Jul 06 22:07:09.515057 ip-10-0-134-202 kubenswrapper[2559]: I0706 22:07:09.515019    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ckj9h_ad24b800-9783-47dc-aaa1-4d2bbc01afb8/kube-rbac-proxy/0.log"