Jul 02 21:18:46.627087 ip-10-0-135-227 systemd[1]: Starting Kubernetes Kubelet...
Jul 02 21:18:47.009773 ip-10-0-135-227 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 21:18:47.009773 ip-10-0-135-227 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 02 21:18:47.009773 ip-10-0-135-227 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 21:18:47.009773 ip-10-0-135-227 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 02 21:18:47.009773 ip-10-0-135-227 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 21:18:47.009773 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.008959    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 02 21:18:47.016932 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014301    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:47.016932 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014313    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:47.016932 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014317    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:47.016932 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014320    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:47.016932 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014326    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:47.016932 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014330    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:47.016932 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014333    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:47.016932 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014335    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:47.016932 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014338    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:47.016932 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014342    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:47.016932 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014344    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:47.016932 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014347    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:47.016932 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014350    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:47.016932 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014353    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:47.016932 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014356    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:47.016932 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014359    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:47.016932 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014362    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:47.016932 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014367    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:47.016932 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014370    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:47.019703 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014373    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:47.019703 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014375    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:47.019703 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014378    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:47.019703 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014381    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:47.019703 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014383    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:47.019703 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014386    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:47.019703 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014389    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:47.019703 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014392    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:47.019703 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014394    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:47.019703 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014397    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:47.019703 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014402    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:47.019703 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014405    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:47.019703 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014408    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:47.019703 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014412    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:47.019703 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014418    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:47.019703 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014421    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:47.019703 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014424    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:47.019703 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014427    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:47.019703 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014430    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:47.020333 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014435    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:47.020333 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014439    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:47.020333 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014445    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:47.020333 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014448    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:47.020333 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014451    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:47.020333 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014454    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:47.020333 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014457    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:47.020333 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014460    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:47.020333 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014463    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:47.020333 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014466    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:47.020333 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014469    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:47.020333 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014472    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:47.020333 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014475    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:47.020333 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014478    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:47.020333 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014480    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:47.020333 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014485    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:47.020333 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014488    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:47.020333 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014491    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:47.020333 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014493    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:47.023119 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014496    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:47.023119 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014499    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:47.023119 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014501    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:47.023119 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014504    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:47.023119 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014506    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:47.023119 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014509    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:47.023119 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014512    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:47.023119 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014514    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:47.023119 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014518    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:47.023119 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014523    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:47.023119 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014525    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:47.023119 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014528    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:47.023119 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014531    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:47.023119 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014536    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:47.023119 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014538    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:47.023119 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014541    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:47.023119 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014545    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:47.023119 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014549    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:47.023119 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014552    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:47.023119 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014555    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:47.023759 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014560    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:47.023759 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014563    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:47.023759 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014565    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:47.023759 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014569    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:47.023759 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014571    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:47.023759 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014574    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:47.023759 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014576    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:47.023759 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014579    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:47.023759 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014582    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:47.023759 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014585    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:47.023759 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014587    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:47.023759 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014590    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:47.023759 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014593    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:47.023759 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014598    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:47.023759 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014600    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:47.023759 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014603    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:47.023759 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014606    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:47.023759 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014608    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:47.023759 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014611    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:47.023759 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014614    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:47.026585 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014617    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:47.026585 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.014620    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:47.111640 ip-10-0-135-227 systemd[1]: Started Kubernetes Kubelet.
Jul 02 21:18:47.117259 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043705    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:47.117259 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043717    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:47.117259 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043721    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:47.117259 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043723    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:47.117259 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043727    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:47.117259 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043730    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:47.117259 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043733    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:47.117259 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043738    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:47.117259 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043742    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:47.117259 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043746    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:47.117259 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043748    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:47.117259 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043752    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:47.117259 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043755    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:47.117259 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043757    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:47.117259 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043760    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:47.117259 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043763    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:47.117259 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043765    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:47.117259 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043768    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:47.117259 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043771    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:47.119477 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043773    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:47.119477 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043790    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:47.119477 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043794    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:47.119477 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043796    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:47.119477 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043799    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:47.119477 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043802    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:47.119477 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043804    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:47.119477 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043807    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:47.119477 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043809    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:47.119477 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043812    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:47.119477 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043815    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:47.119477 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043818    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:47.119477 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043821    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:47.119477 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043824    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:47.119477 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043828    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:47.119477 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043830    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:47.119477 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043833    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:47.119477 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043835    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:47.119477 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043839    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:47.119477 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043841    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:47.120554 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043843    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:47.120554 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043846    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:47.120554 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043848    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:47.120554 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043851    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:47.120554 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043853    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:47.120554 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043856    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:47.120554 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043858    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:47.120554 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043861    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:47.120554 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043864    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:47.120554 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043866    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:47.120554 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043869    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:47.120554 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043873    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:47.120554 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043876    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:47.120554 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043878    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:47.120554 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043880    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:47.120554 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043883    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:47.120554 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043885    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:47.120554 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043888    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:47.120554 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043890    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:47.120554 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043893    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:47.123398 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043895    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:47.123398 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043898    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:47.123398 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043902    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:47.123398 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043905    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:47.123398 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043907    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:47.123398 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043910    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:47.123398 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043913    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:47.123398 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043915    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:47.123398 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043918    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:47.123398 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043922    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:47.123398 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043926    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:47.123398 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043928    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:47.123398 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043931    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:47.123398 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043935    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:47.123398 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043937    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:47.123398 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043940    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:47.123398 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043942    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:47.123398 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043945    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:47.123398 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043948    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:47.124274 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043950    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:47.124274 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043953    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:47.124274 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043955    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:47.124274 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043958    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:47.124274 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043960    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:47.124274 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043964    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:47.124274 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043967    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:47.124274 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043969    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:47.124274 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043972    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:47.124274 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043974    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:47.124274 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043977    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:47.124274 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043980    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:47.124274 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043982    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:47.124274 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043984    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:47.124274 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043987    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:47.124274 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043990    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:47.124274 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043992    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:47.124274 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043995    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:47.124274 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.043999    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:47.124274 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044004    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:47.127046 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044007    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:47.127046 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044372    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:47.127046 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044377    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:47.127046 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044380    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:47.127046 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044383    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:47.127046 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044385    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:47.127046 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044390    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:47.127046 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044393    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:47.127046 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044397    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:47.127046 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044401    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:47.127046 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044405    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:47.127046 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044408    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:47.127046 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044411    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:47.127046 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044414    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:47.127046 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044417    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:47.127046 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044419    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:47.127046 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044422    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:47.127046 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044425    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:47.127046 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044428    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:47.127046 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044430    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:47.128165 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044432    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:47.128165 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044435    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:47.128165 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044437    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:47.128165 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044440    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:47.128165 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044444    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:47.128165 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044447    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:47.128165 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044450    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:47.128165 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044453    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:47.128165 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044456    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:47.128165 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044458    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:47.128165 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044461    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:47.128165 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044464    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:47.128165 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044466    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:47.128165 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044469    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:47.128165 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044471    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:47.128165 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044474    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:47.128165 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044477    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:47.128165 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044480    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:47.128165 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044482    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:47.203325 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044485    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:47.203325 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044488    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:47.203325 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044490    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:47.203325 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044493    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:47.203325 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044496    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:47.203325 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044498    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:47.203325 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044501    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:47.203325 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044503    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:47.203325 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044506    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:47.203325 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044508    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:47.203325 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044511    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:47.203325 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044514    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:47.203325 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044516    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:47.203325 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044518    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:47.203325 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044521    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:47.203325 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044523    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:47.203325 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044526    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:47.203325 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044528    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:47.203325 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044531    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:47.203325 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044533    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:47.204641 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044536    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:47.204641 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044539    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:47.204641 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044541    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:47.204641 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044544    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:47.204641 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044547    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:47.204641 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044549    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:47.204641 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044552    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:47.204641 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044554    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:47.204641 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044557    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:47.204641 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044560    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:47.204641 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044562    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:47.204641 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044565    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:47.204641 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044567    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:47.204641 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044569    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:47.204641 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044572    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:47.204641 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044574    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:47.204641 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044577    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:47.204641 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044579    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:47.204641 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044582    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:47.204641 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044584    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:47.205831 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044587    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:47.205831 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044590    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:47.205831 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044594    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:47.205831 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044596    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:47.205831 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044599    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:47.205831 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044601    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:47.205831 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044604    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:47.205831 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044606    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:47.205831 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044609    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:47.205831 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044611    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:47.205831 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044614    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:47.205831 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044617    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:47.205831 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044619    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:47.205831 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044622    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:47.205831 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044624    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:47.205831 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044626    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:47.205831 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044629    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:47.205831 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044632    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:47.205831 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044634    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:47.205831 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044637    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.044639    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044707    2575 flags.go:64] FLAG: --address="0.0.0.0"
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044714    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044720    2575 flags.go:64] FLAG: --anonymous-auth="true"
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044724    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044728    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044731    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044735    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044739    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044742    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044745    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044748    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044751    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044755    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044758    2575 flags.go:64] FLAG: --cgroup-root=""
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044761    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044764    2575 flags.go:64] FLAG: --client-ca-file=""
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044767    2575 flags.go:64] FLAG: --cloud-provider="external"
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044770    2575 flags.go:64] FLAG: --cluster-dns="[]"
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044789    2575 flags.go:64] FLAG: --cluster-domain=""
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044792    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044795    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044798    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 02 21:18:47.207508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044801    2575 flags.go:64] FLAG: --container-log-max-files="5"
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044805    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044808    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044811    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044815    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044818    2575 flags.go:64] FLAG: --contention-profiling="false"
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044821    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044824    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044827    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044830    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044834    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044837    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044840    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044843    2575 flags.go:64] FLAG: --enable-load-reader="false"
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044846    2575 flags.go:64] FLAG: --enable-server="true"
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044849    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044853    2575 flags.go:64] FLAG: --event-burst="100"
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044856    2575 flags.go:64] FLAG: --event-qps="50"
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044859    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044863    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044865    2575 flags.go:64] FLAG: --eviction-hard=""
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044874    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044877    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044880    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044883    2575 flags.go:64] FLAG: --eviction-soft=""
Jul 02 21:18:47.208620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044886    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044889    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044892    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044896    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044899    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044901    2575 flags.go:64] FLAG: --fail-swap-on="true"
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044904    2575 flags.go:64] FLAG: --feature-gates=""
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044908    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044910    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044913    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044916    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044919    2575 flags.go:64] FLAG: --healthz-port="10248"
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044925    2575 flags.go:64] FLAG: --help="false"
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044928    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-135-227.ec2.internal"
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044931    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044934    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044937    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044940    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044944    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044947    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044950    2575 flags.go:64] FLAG: --image-service-endpoint=""
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044953    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044956    2575 flags.go:64] FLAG: --kube-api-burst="100"
Jul 02 21:18:47.210135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044959    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044962    2575 flags.go:64] FLAG: --kube-api-qps="50"
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044965    2575 flags.go:64] FLAG: --kube-reserved=""
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044968    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044970    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044973    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044976    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044979    2575 flags.go:64] FLAG: --lock-file=""
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044982    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044985    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044988    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.044996    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045001    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045004    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045007    2575 flags.go:64] FLAG: --logging-format="text"
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045010    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045013    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045016    2575 flags.go:64] FLAG: --manifest-url=""
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045018    2575 flags.go:64] FLAG: --manifest-url-header=""
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045022    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045026    2575 flags.go:64] FLAG: --max-open-files="1000000"
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045031    2575 flags.go:64] FLAG: --max-pods="110"
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045034    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045037    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045039    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 02 21:18:47.211256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045042    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045045    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045048    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045051    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045057    2575 flags.go:64] FLAG: --node-status-max-images="50"
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045060    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045063    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045067    2575 flags.go:64] FLAG: --pod-cidr=""
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045069    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045074    2575 flags.go:64] FLAG: --pod-manifest-path=""
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045077    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045080    2575 flags.go:64] FLAG: --pods-per-core="0"
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045083    2575 flags.go:64] FLAG: --port="10250"
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045086    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045089    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08c47d5d4c0a43af4"
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045092    2575 flags.go:64] FLAG: --qos-reserved=""
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045095    2575 flags.go:64] FLAG: --read-only-port="10255"
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045098    2575 flags.go:64] FLAG: --register-node="true"
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045101    2575 flags.go:64] FLAG: --register-with-taints=""
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045105    2575 flags.go:64] FLAG: --registry-burst="10"
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045109    2575 flags.go:64] FLAG: --registry-qps="5"
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045112    2575 flags.go:64] FLAG: --reserved-cpus=""
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045114    2575 flags.go:64] FLAG: --reserved-memory=""
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045118    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045121    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 02 21:18:47.212575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045123    2575 flags.go:64] FLAG: --rotate-certificates="false"
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045126    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045129    2575 flags.go:64] FLAG: --runonce="false"
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045132    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045136    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045138    2575 flags.go:64] FLAG: --seccomp-default="false"
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045141    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045144    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045147    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045150    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045153    2575 flags.go:64] FLAG: --storage-driver-password="root"
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045156    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045158    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045161    2575 flags.go:64] FLAG: --storage-driver-user="root"
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045164    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045167    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045170    2575 flags.go:64] FLAG: --system-cgroups=""
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045172    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045177    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045180    2575 flags.go:64] FLAG: --tls-cert-file=""
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045182    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045187    2575 flags.go:64] FLAG: --tls-min-version=""
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045190    2575 flags.go:64] FLAG: --tls-private-key-file=""
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045192    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045195    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 02 21:18:47.214559 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045198    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 02 21:18:47.215499 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045203    2575 flags.go:64] FLAG: --v="2"
Jul 02 21:18:47.215499 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045206    2575 flags.go:64] FLAG: --version="false"
Jul 02 21:18:47.215499 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045212    2575 flags.go:64] FLAG: --vmodule=""
Jul 02 21:18:47.215499 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045215    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 02 21:18:47.215499 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.045219    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 02 21:18:47.215499 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045357    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:47.215499 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045362    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:47.215499 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045365    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:47.215499 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045368    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:47.215499 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045371    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:47.215499 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045375    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:47.215499 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045378    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:47.215499 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045381    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:47.215499 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045384    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:47.215499 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045386    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:47.215499 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045391    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:47.215499 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045395    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:47.215499 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045397    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:47.215499 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045400    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:47.215499 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045403    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:47.216278 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045406    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:47.216278 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045408    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:47.216278 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045411    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:47.216278 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045414    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:47.216278 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045416    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:47.216278 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045419    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:47.216278 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045421    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:47.216278 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045424    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:47.216278 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045426    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:47.216278 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045429    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:47.216278 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045431    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:47.216278 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045434    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:47.216278 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045438    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:47.216278 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045440    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:47.216278 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045443    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:47.216278 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045446    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:47.216278 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045449    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:47.216278 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045451    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:47.216278 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045454    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:47.216278 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045456    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:47.217047 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045458    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:47.217047 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045461    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:47.217047 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045466    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:47.217047 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045468    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:47.217047 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045471    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:47.217047 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045473    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:47.217047 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045476    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:47.217047 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045478    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:47.217047 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045481    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:47.217047 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045484    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:47.217047 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045486    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:47.217047 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045489    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:47.217047 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045491    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:47.217047 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045495    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:47.217047 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045498    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:47.217047 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045501    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:47.217047 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045504    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:47.217047 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045506    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:47.217047 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045509    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:47.217047 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045511    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:47.217765 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045514    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:47.217765 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045516    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:47.217765 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045519    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:47.217765 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045521    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:47.217765 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045527    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:47.217765 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045530    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:47.217765 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045534    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:47.217765 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045537    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:47.217765 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045541    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:47.217765 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045544    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:47.217765 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045547    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:47.217765 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045550    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:47.217765 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045553    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:47.217765 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045556    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:47.217765 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045560    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:47.217765 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045563    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:47.217765 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045566    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:47.217765 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045568    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:47.217765 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045571    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:47.217765 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045574    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:47.218510 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045577    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:47.218510 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045580    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:47.218510 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045582    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:47.218510 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045585    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:47.218510 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045587    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:47.218510 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045590    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:47.218510 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045592    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:47.218510 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045595    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:47.218510 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045597    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:47.218510 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045600    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:47.218510 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045602    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:47.218510 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045605    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:47.218510 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045607    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:47.218510 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045610    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:47.218510 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045613    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:47.218510 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045615    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:47.218510 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045619    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:47.218510 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045622    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:47.218510 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045625    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:47.218510 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045627    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:47.219238 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045630    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:47.219238 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045633    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:47.219238 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045635    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:47.219238 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.045638    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:47.219238 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.046406    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 21:18:47.219238 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.061282    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 02 21:18:47.219238 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.061299    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 02 21:18:47.219238 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.061993    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 02 21:18:47.219238 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.062004    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 02 21:18:47.219238 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062077    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:47.219238 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062083    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:47.219238 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062086    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:47.219238 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062090    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:47.219238 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062092    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:47.219238 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062095    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:47.219803 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062098    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:47.219803 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062101    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:47.219803 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062104    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:47.219803 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062106    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:47.219803 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062109    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:47.219803 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062112    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:47.219803 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062115    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:47.219803 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062118    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:47.219803 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062120    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:47.219803 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062123    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:47.219803 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062125    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:47.219803 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062129    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:47.219803 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062131    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:47.219803 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062134    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:47.219803 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062137    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:47.219803 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062140    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:47.219803 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062142    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:47.219803 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062147    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:47.219803 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062150    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:47.219803 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062154    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:47.220523 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062157    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:47.220523 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062159    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:47.220523 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062172    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:47.220523 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062176    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:47.220523 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062178    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:47.220523 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062181    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:47.220523 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062183    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:47.220523 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062186    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:47.220523 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062189    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:47.220523 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062192    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:47.220523 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062195    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:47.220523 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062197    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:47.220523 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062200    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:47.220523 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062202    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:47.220523 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062205    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:47.220523 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062208    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:47.220523 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062210    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:47.220523 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062213    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:47.220523 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062216    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:47.220523 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062218    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:47.221244 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062221    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:47.221244 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062223    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:47.221244 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062226    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:47.221244 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062228    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:47.221244 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062231    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:47.221244 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062234    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:47.221244 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062237    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:47.221244 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062240    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:47.221244 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062244    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:47.221244 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062247    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:47.221244 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062250    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:47.221244 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062254    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:47.221244 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062256    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:47.221244 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062259    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:47.221244 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062262    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:47.221244 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062265    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:47.221244 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062268    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:47.221244 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062271    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:47.221244 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062273    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:47.221938 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062276    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:47.221938 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062279    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:47.221938 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062282    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:47.221938 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062284    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:47.221938 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062287    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:47.221938 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062289    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:47.221938 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062292    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:47.221938 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062295    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:47.221938 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062299    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:47.221938 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062302    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:47.221938 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062306    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:47.221938 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062309    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:47.221938 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062312    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:47.221938 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062314    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:47.221938 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062317    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:47.221938 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062320    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:47.221938 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062323    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:47.221938 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062325    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:47.221938 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062328    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:47.222624 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062330    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:47.222624 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062333    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:47.222624 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062335    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:47.222624 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062337    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:47.222624 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062340    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:47.222624 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062344    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:47.222624 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062347    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:47.222624 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062350    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:47.222624 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062352    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:47.222624 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062355    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:47.222624 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062357    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:47.222624 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062361    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:47.222624 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062363    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:47.222624 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062366    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:47.222624 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062368    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:47.223177 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.062374    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 21:18:47.223177 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062504    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:47.223177 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062509    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:47.223177 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062512    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:47.223177 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062515    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:47.223177 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062518    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:47.223177 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062520    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:47.223177 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062523    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:47.223177 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062527    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:47.223177 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062529    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:47.223177 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062532    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:47.223177 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062535    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:47.223177 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062537    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:47.223177 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062540    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:47.223177 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062542    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:47.223717 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062545    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:47.223717 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062548    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:47.223717 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062551    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:47.223717 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062553    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:47.223717 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062556    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:47.223717 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062558    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:47.223717 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062561    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:47.223717 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062563    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:47.223717 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062567    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:47.223717 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062569    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:47.223717 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062572    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:47.223717 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062575    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:47.223717 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062577    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:47.223717 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062580    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:47.223717 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062583    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:47.223717 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062585    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:47.223717 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062588    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:47.223717 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062592    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:47.223717 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062595    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:47.223717 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062598    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:47.224448 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062601    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:47.224448 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062603    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:47.224448 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062607    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:47.224448 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062610    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:47.224448 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062612    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:47.224448 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062616    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:47.224448 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062619    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:47.224448 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062621    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:47.224448 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062624    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:47.224448 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062626    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:47.224448 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062629    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:47.224448 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062631    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:47.224448 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062634    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:47.224448 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062636    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:47.224448 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062639    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:47.224448 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062641    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:47.224448 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062644    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:47.224448 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062647    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:47.224448 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062649    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:47.224448 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062652    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:47.225180 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062655    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:47.225180 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062658    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:47.225180 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062661    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:47.225180 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062665    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:47.225180 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062668    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:47.225180 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062670    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:47.225180 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062673    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:47.225180 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062676    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:47.225180 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062679    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:47.225180 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062681    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:47.225180 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062684    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:47.225180 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062686    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:47.225180 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062688    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:47.225180 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062691    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:47.225180 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062693    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:47.225180 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062696    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:47.225180 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062698    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:47.225180 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062702    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:47.225180 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062705    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:47.225859 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062707    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:47.225859 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062710    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:47.225859 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062712    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:47.225859 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062715    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:47.225859 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062718    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:47.225859 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062721    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:47.225859 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062723    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:47.225859 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062726    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:47.225859 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062728    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:47.225859 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062730    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:47.225859 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062733    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:47.225859 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062736    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:47.225859 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062738    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:47.225859 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062742    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:47.225859 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062745    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:47.225859 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062748    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:47.225859 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062751    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:47.225859 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062753    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:47.225859 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062756    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:47.225859 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062758    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:47.226568 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062761    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:47.226568 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062764    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:47.226568 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062768    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:47.226568 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062771    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:47.226568 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062789    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:47.226568 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:47.062793    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:47.226568 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.062799    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 21:18:47.226568 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.062901    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 02 21:18:47.226568 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.064982    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 02 21:18:47.226568 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.066362    2575 server.go:1019] "Starting client certificate rotation"
Jul 02 21:18:47.226568 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.066462    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 21:18:47.226568 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.066493    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 21:18:47.226568 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.068754    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 21:18:47.227043 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.071075    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 21:18:47.227043 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.081960    2575 log.go:25] "Validated CRI v1 runtime API"
Jul 02 21:18:47.227043 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.083995    2575 log.go:25] "Validated CRI v1 image API"
Jul 02 21:18:47.227043 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.084627    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 02 21:18:47.227043 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.086840    2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a91f323d-1a16-4897-b82a-aa07cab9f12d:/dev/nvme0n1p3 b9eff7a4-56ff-4971-9744-5ceddb8eb6cd:/dev/nvme0n1p4]
Jul 02 21:18:47.227043 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.086854    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 02 21:18:47.227272 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.091513    2575 manager.go:216] Machine: {Timestamp:2026-07-02 21:18:47.090349513 +0000 UTC m=+0.349241718 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108584 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2eee1e84e09fba87cb0a4033422d18 SystemUUID:ec2eee1e-84e0-9fba-87cb-0a4033422d18 BootID:cfaad660-a760-4060-bc03-046376efd67d Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fe:dd:b5:6a:fb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fe:dd:b5:6a:fb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:03:52:75:7d:d0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 02 21:18:47.227272 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.091612    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 02 21:18:47.227272 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.091679    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 02 21:18:47.227272 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.093261    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 02 21:18:47.227272 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.093284    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-227.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 02 21:18:47.227272 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.093922    2575 topology_manager.go:138] "Creating topology manager with none policy"
Jul 02 21:18:47.227272 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.093932    2575 container_manager_linux.go:309] "Creating device plugin manager"
Jul 02 21:18:47.227272 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.093940    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 21:18:47.227272 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.094521    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 21:18:47.227272 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.095821    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 02 21:18:47.227272 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.098614    2575 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 21:18:47.227272 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.098706    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 21:18:47.227272 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.098791    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 02 21:18:47.227272 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.098823    2575 kubelet.go:486] "Attempting to sync node with API server"
Jul 02 21:18:47.227272 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.098834    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 02 21:18:47.227272 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.098849    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 02 21:18:47.227272 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.098860    2575 kubelet.go:398] "Adding apiserver pod source"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.098878    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.099223    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.099239    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.101268    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.103039    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.103068    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.104112    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.104135    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.104142    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.104149    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.104158    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.104163    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.104171    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.104176    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.104182    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.104187    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.104203    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.104209    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.104836    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.104846    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.107993    2575 server.go:1268] "Started kubelet"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.108070    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.111927    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-227.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:47.112018    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:47.112310    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-227.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.108070    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.112691    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.114234    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:47.115765    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.116745    2575 server.go:310] "Adding debug handlers to kubelet server"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:47.117061    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-227.ec2.internal.18be95b9a827de97  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-227.ec2.internal,UID:ip-10-0-135-227.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-227.ec2.internal,},FirstTimestamp:2026-07-02 21:18:47.107968663 +0000 UTC m=+0.366860867,LastTimestamp:2026-07-02 21:18:47.107968663 +0000 UTC m=+0.366860867,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-227.ec2.internal,}"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.119424    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.119411    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.121002    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:47.121074    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-227.ec2.internal\" not found"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.121751    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.121765    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.121854    2575 reconstruct.go:97] "Volume reconstruction finished"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.121861    2575 reconciler.go:29] "Reconciler: start to sync state"
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.124264    2575 factory.go:55] Registering systemd factory
Jul 02 21:18:47.227934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.124284    2575 factory.go:223] Registration of the systemd container factory successfully
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.124633    2575 factory.go:153] Registering CRI-O factory
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.124646    2575 factory.go:223] Registration of the crio container factory successfully
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.124743    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.124790    2575 factory.go:103] Registering Raw factory
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.124813    2575 manager.go:1196] Started watching for new ooms in manager
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.125744    2575 manager.go:349] Starting recovery of all containers
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:47.129946    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:47.130050    2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-227.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.148000    2575 manager.go:354] Recovery completed
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.151552    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.153261    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.153285    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.153297    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.153888    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.153896    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.153914    2575 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.154208    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.154837    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.154852    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.154865    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.155196    2575 policy_none.go:49] "None policy: Start"
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.155213    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.155225    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.156427    2575 policy_none.go:47] "Start"
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.158123    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ccp2k"
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:47.160057    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-227.ec2.internal.18be95b9aadb3491  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-227.ec2.internal,UID:ip-10-0-135-227.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-227.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-227.ec2.internal,},FirstTimestamp:2026-07-02 21:18:47.153276049 +0000 UTC m=+0.412168254,LastTimestamp:2026-07-02 21:18:47.153276049 +0000 UTC m=+0.412168254,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-227.ec2.internal,}"
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.166119    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ccp2k"
Jul 02 21:18:47.229437 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:47.169673    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-227.ec2.internal.18be95b9aadb6baa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-227.ec2.internal,UID:ip-10-0-135-227.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-227.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-227.ec2.internal,},FirstTimestamp:2026-07-02 21:18:47.153290154 +0000 UTC m=+0.412182359,LastTimestamp:2026-07-02 21:18:47.153290154 +0000 UTC m=+0.412182359,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-227.ec2.internal,}"
Jul 02 21:18:47.230542 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:47.178703    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-227.ec2.internal.18be95b9aadb96d3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-227.ec2.internal,UID:ip-10-0-135-227.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-227.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-227.ec2.internal,},FirstTimestamp:2026-07-02 21:18:47.153301203 +0000 UTC m=+0.412193407,LastTimestamp:2026-07-02 21:18:47.153301203 +0000 UTC m=+0.412193407,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-227.ec2.internal,}"
Jul 02 21:18:47.230542 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:47.221174    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-227.ec2.internal\" not found"
Jul 02 21:18:47.247199 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.247151    2575 manager.go:337] "Starting Device Plugin manager"
Jul 02 21:18:47.247199 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:47.247183    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 02 21:18:47.247199 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.247192    2575 server.go:87] "Starting device plugin registration server"
Jul 02 21:18:47.247449 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.247438    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 02 21:18:47.247505 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.247452    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 02 21:18:47.247549 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.247533    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 02 21:18:47.247672 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.247636    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 02 21:18:47.247672 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.247648    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 02 21:18:47.248170 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:47.248149    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 02 21:18:47.248243 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:47.248194    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-227.ec2.internal\" not found"
Jul 02 21:18:47.336063 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:47.336040    2575 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-227.ec2.internal\" not found" node="ip-10-0-135-227.ec2.internal"
Jul 02 21:18:47.348282 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.348267    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:47.349123 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.349110    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:47.349176 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.349136    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:47.349176 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.349148    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:47.349241 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.349179    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-227.ec2.internal"
Jul 02 21:18:47.359414 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.359399    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-227.ec2.internal"
Jul 02 21:18:47.359479 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:47.359418    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-227.ec2.internal\": node \"ip-10-0-135-227.ec2.internal\" not found"
Jul 02 21:18:47.374803 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.374767    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 02 21:18:47.375945 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.375926    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 02 21:18:47.376025 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.375957    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 02 21:18:47.376025 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.375974    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 02 21:18:47.376025 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:47.376006    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 02 21:18:47.378617 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:47.378598    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:18:47.389519 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:47.389503    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-227.ec2.internal\" not found"
Jul 02 21:18:49.109246 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.109215    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-227.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-227.ec2.internal"]
Jul 02 21:18:49.109320 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.109293    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:49.109980 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.109964    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:49.110034 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.109991    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:49.110034 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.110001    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:49.112051 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.112039    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:49.112159 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.112145    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-227.ec2.internal"
Jul 02 21:18:49.112222 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.112186    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:49.112820 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.112804    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:49.112820 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.112812    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:49.112820 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.112832    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:49.113020 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.112847    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:49.113020 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.112835    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:49.113020 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.112924    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:49.114758 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.114743    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-227.ec2.internal"
Jul 02 21:18:49.114847 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.114775    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:49.115407 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.115392    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:49.115468 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.115418    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:49.115468 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.115427    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:49.122425 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:49.122411    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-227.ec2.internal\" not found"
Jul 02 21:18:49.129738 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:49.129695    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-227.ec2.internal\" not found" node="ip-10-0-135-227.ec2.internal"
Jul 02 21:18:49.133638 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:49.133624    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-227.ec2.internal\" not found" node="ip-10-0-135-227.ec2.internal"
Jul 02 21:18:49.156873 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.156855    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/912e0ce5f47be8c71daf0c28291ccddf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-227.ec2.internal\" (UID: \"912e0ce5f47be8c71daf0c28291ccddf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-227.ec2.internal"
Jul 02 21:18:49.156933 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.156878    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/912e0ce5f47be8c71daf0c28291ccddf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-227.ec2.internal\" (UID: \"912e0ce5f47be8c71daf0c28291ccddf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-227.ec2.internal"
Jul 02 21:18:49.156933 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.156894    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c947105a64a6be554217ac8fde87371e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-227.ec2.internal\" (UID: \"c947105a64a6be554217ac8fde87371e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-227.ec2.internal"
Jul 02 21:18:49.223492 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:49.223471    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-227.ec2.internal\" not found"
Jul 02 21:18:49.257235 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.257214    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/912e0ce5f47be8c71daf0c28291ccddf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-227.ec2.internal\" (UID: \"912e0ce5f47be8c71daf0c28291ccddf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-227.ec2.internal"
Jul 02 21:18:49.257327 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.257237    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c947105a64a6be554217ac8fde87371e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-227.ec2.internal\" (UID: \"c947105a64a6be554217ac8fde87371e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-227.ec2.internal"
Jul 02 21:18:49.257327 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.257252    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/912e0ce5f47be8c71daf0c28291ccddf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-227.ec2.internal\" (UID: \"912e0ce5f47be8c71daf0c28291ccddf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-227.ec2.internal"
Jul 02 21:18:49.257327 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.257277    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/912e0ce5f47be8c71daf0c28291ccddf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-227.ec2.internal\" (UID: \"912e0ce5f47be8c71daf0c28291ccddf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-227.ec2.internal"
Jul 02 21:18:49.257327 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.257308    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c947105a64a6be554217ac8fde87371e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-227.ec2.internal\" (UID: \"c947105a64a6be554217ac8fde87371e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-227.ec2.internal"
Jul 02 21:18:49.257327 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.257309    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/912e0ce5f47be8c71daf0c28291ccddf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-227.ec2.internal\" (UID: \"912e0ce5f47be8c71daf0c28291ccddf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-227.ec2.internal"
Jul 02 21:18:49.323963 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:49.323941    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-227.ec2.internal\" not found"
Jul 02 21:18:49.424452 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:49.424383    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-227.ec2.internal\" not found"
Jul 02 21:18:49.432558 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.432536    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-227.ec2.internal"
Jul 02 21:18:49.437039 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.436943    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-227.ec2.internal"
Jul 02 21:18:49.524816 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:49.524790    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-227.ec2.internal\" not found"
Jul 02 21:18:49.625234 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:49.625216    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-227.ec2.internal\" not found"
Jul 02 21:18:49.698709 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.698658    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 02 21:18:49.699205 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.698780    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 21:18:49.725959 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:49.725929    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-227.ec2.internal\" not found"
Jul 02 21:18:49.786257 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.786234    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:18:49.816777 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.816747    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 21:13:48 +0000 UTC" deadline="2028-04-01 01:07:45.644588962 +0000 UTC"
Jul 02 21:18:49.816777 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.816772    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15315h48m55.827820183s"
Jul 02 21:18:49.826899 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:49.826881    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-227.ec2.internal\" not found"
Jul 02 21:18:49.873156 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.873141    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:18:49.922435 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.922417    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:49.923606 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.923591    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:49.923682 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.923618    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:49.923682 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:49.923628    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:49.927685 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:49.927666    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-227.ec2.internal\" not found"
Jul 02 21:18:49.996094 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:49.996069    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc947105a64a6be554217ac8fde87371e.slice/crio-d831272a78257ddf5c3fae92fc20543ae9a21cb8f29860ee585f0f9a89de4f12 WatchSource:0}: Error finding container d831272a78257ddf5c3fae92fc20543ae9a21cb8f29860ee585f0f9a89de4f12: Status 404 returned error can't find the container with id d831272a78257ddf5c3fae92fc20543ae9a21cb8f29860ee585f0f9a89de4f12
Jul 02 21:18:49.996343 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:49.996321    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912e0ce5f47be8c71daf0c28291ccddf.slice/crio-3cc2844f6e42ccfe5cacebbdf93fc66c41b55d407ea9e86cdbeb82f9093377c5 WatchSource:0}: Error finding container 3cc2844f6e42ccfe5cacebbdf93fc66c41b55d407ea9e86cdbeb82f9093377c5: Status 404 returned error can't find the container with id 3cc2844f6e42ccfe5cacebbdf93fc66c41b55d407ea9e86cdbeb82f9093377c5
Jul 02 21:18:50.000249 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.000234    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 21:18:50.011482 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.011442    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-227.ec2.internal" event={"ID":"c947105a64a6be554217ac8fde87371e","Type":"ContainerStarted","Data":"d831272a78257ddf5c3fae92fc20543ae9a21cb8f29860ee585f0f9a89de4f12"}
Jul 02 21:18:50.012474 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.012453    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-227.ec2.internal" event={"ID":"912e0ce5f47be8c71daf0c28291ccddf","Type":"ContainerStarted","Data":"3cc2844f6e42ccfe5cacebbdf93fc66c41b55d407ea9e86cdbeb82f9093377c5"}
Jul 02 21:18:50.028383 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:50.028356    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-227.ec2.internal\" not found"
Jul 02 21:18:50.128831 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:50.128805    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-227.ec2.internal\" not found"
Jul 02 21:18:50.229294 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:50.229275    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-227.ec2.internal\" not found"
Jul 02 21:18:50.234974 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.234960    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:18:50.254202 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.254141    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-227.ec2.internal"
Jul 02 21:18:50.264334 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.264318    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 21:18:50.265299 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.265287    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-227.ec2.internal"
Jul 02 21:18:50.277962 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.277946    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 21:18:50.732803 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.732721    2575 apiserver.go:52] "Watching apiserver"
Jul 02 21:18:50.744291 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.744239    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 02 21:18:50.747284 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.746935    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-pxcm5","openshift-network-diagnostics/network-check-target-b9tf7","openshift-network-operator/iptables-alerter-qkjh4","kube-system/konnectivity-agent-dnrb6","kube-system/kube-apiserver-proxy-ip-10-0-135-227.ec2.internal","openshift-cluster-node-tuning-operator/tuned-bb2r6","openshift-dns/node-resolver-k4r2h","openshift-multus/multus-w9fcp","openshift-ovn-kubernetes/ovnkube-node-2725k","kube-system/global-pull-secret-syncer-gpccf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n","openshift-image-registry/node-ca-9zmmn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-227.ec2.internal","openshift-multus/multus-additional-cni-plugins-rfbk7"]
Jul 02 21:18:50.749966 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.749848    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k4r2h"
Jul 02 21:18:50.751985 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.751930    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.752402 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.752383    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 02 21:18:50.753280 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.753248    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v4c28\""
Jul 02 21:18:50.753530 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.753513    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 02 21:18:50.753693 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.753677    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 02 21:18:50.756322 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.754727    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 02 21:18:50.756322 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.754941    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xfbnc\""
Jul 02 21:18:50.756322 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.755210    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 02 21:18:50.756322 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.755398    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 02 21:18:50.756322 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.755729    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 02 21:18:50.756932 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.756727    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qkjh4"
Jul 02 21:18:50.757120 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.757100    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dnrb6"
Jul 02 21:18:50.759437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.759327    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:18:50.759526 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:50.759442    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxcm5" podUID="28a7f381-3db6-43e5-9690-d409f048dcf2"
Jul 02 21:18:50.760599 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.760580    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 02 21:18:50.761571 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.760955    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 02 21:18:50.761571 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.761211    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nw4wl\""
Jul 02 21:18:50.761571 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.761411    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 02 21:18:50.762506 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.761926    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.762506 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.762123    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 02 21:18:50.762506 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.762381    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cbxjt\""
Jul 02 21:18:50.763407 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.763365    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 02 21:18:50.763898 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.763880    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.764811 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.764795    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 02 21:18:50.764944 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.764806    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 02 21:18:50.765261 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.764806    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-djhn7\""
Jul 02 21:18:50.765261 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.764903    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-host-run-netns\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.765261 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.765109    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-host-var-lib-kubelet\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.765261 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.765140    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c7410f-98e2-4331-b24b-6eeab7f99401-tmp-dir\") pod \"node-resolver-k4r2h\" (UID: \"a0c7410f-98e2-4331-b24b-6eeab7f99401\") " pod="openshift-dns/node-resolver-k4r2h"
Jul 02 21:18:50.765261 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.765193    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vdcg\" (UniqueName: \"kubernetes.io/projected/a0c7410f-98e2-4331-b24b-6eeab7f99401-kube-api-access-5vdcg\") pod \"node-resolver-k4r2h\" (UID: \"a0c7410f-98e2-4331-b24b-6eeab7f99401\") " pod="openshift-dns/node-resolver-k4r2h"
Jul 02 21:18:50.765261 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.765232    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-system-cni-dir\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.765261 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.765256    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-os-release\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.765683 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.765278    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-host-run-k8s-cni-cncf-io\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.765683 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.765301    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-host-var-lib-cni-multus\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.765683 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.765322    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-hostroot\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.765683 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.765346    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc228c55-194a-429b-a1dd-75001e9b6750-cni-binary-copy\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.765683 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.765368    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-multus-conf-dir\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.765683 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.765391    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dc228c55-194a-429b-a1dd-75001e9b6750-multus-daemon-config\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.765683 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.765502    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-host-run-multus-certs\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.765683 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.765526    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-etc-kubernetes\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.765683 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.765549    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84h8t\" (UniqueName: \"kubernetes.io/projected/dc228c55-194a-429b-a1dd-75001e9b6750-kube-api-access-84h8t\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.765683 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.765578    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3dbcf436-a3ce-467a-b081-2cfbc20d3af5-iptables-alerter-script\") pod \"iptables-alerter-qkjh4\" (UID: \"3dbcf436-a3ce-467a-b081-2cfbc20d3af5\") " pod="openshift-network-operator/iptables-alerter-qkjh4"
Jul 02 21:18:50.765683 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.765601    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3dbcf436-a3ce-467a-b081-2cfbc20d3af5-host-slash\") pod \"iptables-alerter-qkjh4\" (UID: \"3dbcf436-a3ce-467a-b081-2cfbc20d3af5\") " pod="openshift-network-operator/iptables-alerter-qkjh4"
Jul 02 21:18:50.765683 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.765624    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a0c7410f-98e2-4331-b24b-6eeab7f99401-hosts-file\") pod \"node-resolver-k4r2h\" (UID: \"a0c7410f-98e2-4331-b24b-6eeab7f99401\") " pod="openshift-dns/node-resolver-k4r2h"
Jul 02 21:18:50.765683 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.765652    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-cnibin\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.765683 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.765675    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-multus-socket-dir-parent\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.766379 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.765708    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-host-var-lib-cni-bin\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.766379 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.765731    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn6kr\" (UniqueName: \"kubernetes.io/projected/3dbcf436-a3ce-467a-b081-2cfbc20d3af5-kube-api-access-zn6kr\") pod \"iptables-alerter-qkjh4\" (UID: \"3dbcf436-a3ce-467a-b081-2cfbc20d3af5\") " pod="openshift-network-operator/iptables-alerter-qkjh4"
Jul 02 21:18:50.766379 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.765756    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-multus-cni-dir\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.766916 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.766776    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 02 21:18:50.766916 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.766853    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.767563 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.767304    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6jrwq\""
Jul 02 21:18:50.767563 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.767378    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 02 21:18:50.767699 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.767668    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 02 21:18:50.769232 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.768777    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 21:18:50.769232 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.768980    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 02 21:18:50.769232 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.768986    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 02 21:18:50.769232 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.769018    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 02 21:18:50.769995 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.769603    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s4csq\""
Jul 02 21:18:50.769995 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.769772    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:18:50.769995 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.769818    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 02 21:18:50.769995 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:50.769833    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9tf7" podUID="4076e581-5038-4786-9565-c72be1618034"
Jul 02 21:18:50.769995 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.769990    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 02 21:18:50.772849 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.772834    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:18:50.773231 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:50.772900    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gpccf" podUID="3a6e6de1-8b1b-478e-8dce-996e81502c1e"
Jul 02 21:18:50.774978 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.774954    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:50.778079 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.778057    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9zmmn"
Jul 02 21:18:50.780137 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.778718    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 02 21:18:50.780137 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.778831    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gv8bp\""
Jul 02 21:18:50.780137 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.778967    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 02 21:18:50.780137 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.779046    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 02 21:18:50.780915 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.780892    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 02 21:18:50.781058 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.781038    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 02 21:18:50.781391 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.781161    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 02 21:18:50.781814 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.781780    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-55xtn\""
Jul 02 21:18:50.793335 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.793313    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-q6sw9"
Jul 02 21:18:50.801163 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.801143    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q6sw9"
Jul 02 21:18:50.854778 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.854752    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 02 21:18:50.866904 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.866867    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-host-run-ovn-kubernetes\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.867059 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.866908    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-multus-cni-dir\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.867059 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.866958    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-host-run-netns\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.867059 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.866995    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dda210c5-c607-473b-b4c9-7d97094ef2dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9rr6n\" (UID: \"dda210c5-c607-473b-b4c9-7d97094ef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:50.867059 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867022    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-run-ovn\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.867059 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867044    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-host-run-netns\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.867348 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867050    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vdcg\" (UniqueName: \"kubernetes.io/projected/a0c7410f-98e2-4331-b24b-6eeab7f99401-kube-api-access-5vdcg\") pod \"node-resolver-k4r2h\" (UID: \"a0c7410f-98e2-4331-b24b-6eeab7f99401\") " pod="openshift-dns/node-resolver-k4r2h"
Jul 02 21:18:50.867348 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867093    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3dbcf436-a3ce-467a-b081-2cfbc20d3af5-iptables-alerter-script\") pod \"iptables-alerter-qkjh4\" (UID: \"3dbcf436-a3ce-467a-b081-2cfbc20d3af5\") " pod="openshift-network-operator/iptables-alerter-qkjh4"
Jul 02 21:18:50.867348 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867101    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-multus-cni-dir\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.867348 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867118    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/83c0ff63-726d-4bb4-b4f6-0810ef1af2b2-serviceca\") pod \"node-ca-9zmmn\" (UID: \"83c0ff63-726d-4bb4-b4f6-0810ef1af2b2\") " pod="openshift-image-registry/node-ca-9zmmn"
Jul 02 21:18:50.867348 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867145    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-host-run-netns\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.867348 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867195    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-var-lib-openvswitch\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.867348 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867225    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc228c55-194a-429b-a1dd-75001e9b6750-cni-binary-copy\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.867348 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867250    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84h8t\" (UniqueName: \"kubernetes.io/projected/dc228c55-194a-429b-a1dd-75001e9b6750-kube-api-access-84h8t\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.867348 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867272    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-run-systemd\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.867348 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867294    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf253477-5a39-4648-9b18-4282286ab281-env-overrides\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.867348 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867318    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-var-lib-kubelet\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.867348 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867343    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vg5d\" (UniqueName: \"kubernetes.io/projected/4b7e06c9-44c9-486b-8483-d71693ccc112-kube-api-access-4vg5d\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.867890 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867371    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-host-var-lib-cni-bin\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.867890 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867395    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4b7e06c9-44c9-486b-8483-d71693ccc112-os-release\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.867890 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867417    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-systemd-units\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.867890 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867439    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-node-log\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.867890 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867462    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf253477-5a39-4648-9b18-4282286ab281-ovnkube-config\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.867890 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867488    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-etc-sysconfig\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.867890 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867509    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-run\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.867890 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867534    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b7e06c9-44c9-486b-8483-d71693ccc112-cni-binary-copy\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.867890 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867605    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b7e06c9-44c9-486b-8483-d71693ccc112-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.867890 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867614    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-host-var-lib-cni-bin\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.867890 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867659    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dda210c5-c607-473b-b4c9-7d97094ef2dd-sys-fs\") pod \"aws-ebs-csi-driver-node-9rr6n\" (UID: \"dda210c5-c607-473b-b4c9-7d97094ef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:50.867890 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867686    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxmd4\" (UniqueName: \"kubernetes.io/projected/83c0ff63-726d-4bb4-b4f6-0810ef1af2b2-kube-api-access-pxmd4\") pod \"node-ca-9zmmn\" (UID: \"83c0ff63-726d-4bb4-b4f6-0810ef1af2b2\") " pod="openshift-image-registry/node-ca-9zmmn"
Jul 02 21:18:50.867890 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867721    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-run-openvswitch\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.867890 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867749    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3dbcf436-a3ce-467a-b081-2cfbc20d3af5-iptables-alerter-script\") pod \"iptables-alerter-qkjh4\" (UID: \"3dbcf436-a3ce-467a-b081-2cfbc20d3af5\") " pod="openshift-network-operator/iptables-alerter-qkjh4"
Jul 02 21:18:50.867890 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867757    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.867890 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867784    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjksx\" (UniqueName: \"kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx\") pod \"network-check-target-b9tf7\" (UID: \"4076e581-5038-4786-9565-c72be1618034\") " pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:18:50.868756 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867794    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc228c55-194a-429b-a1dd-75001e9b6750-cni-binary-copy\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.868756 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867815    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b7e06c9-44c9-486b-8483-d71693ccc112-cnibin\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.868756 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867868    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3dbcf436-a3ce-467a-b081-2cfbc20d3af5-host-slash\") pod \"iptables-alerter-qkjh4\" (UID: \"3dbcf436-a3ce-467a-b081-2cfbc20d3af5\") " pod="openshift-network-operator/iptables-alerter-qkjh4"
Jul 02 21:18:50.868756 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867889    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-etc-modprobe-d\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.868756 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867921    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b7e06c9-44c9-486b-8483-d71693ccc112-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.868756 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867938    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3dbcf436-a3ce-467a-b081-2cfbc20d3af5-host-slash\") pod \"iptables-alerter-qkjh4\" (UID: \"3dbcf436-a3ce-467a-b081-2cfbc20d3af5\") " pod="openshift-network-operator/iptables-alerter-qkjh4"
Jul 02 21:18:50.868756 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.867981    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a0c7410f-98e2-4331-b24b-6eeab7f99401-hosts-file\") pod \"node-resolver-k4r2h\" (UID: \"a0c7410f-98e2-4331-b24b-6eeab7f99401\") " pod="openshift-dns/node-resolver-k4r2h"
Jul 02 21:18:50.868756 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868014    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-cnibin\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.868756 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868052    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a0c7410f-98e2-4331-b24b-6eeab7f99401-hosts-file\") pod \"node-resolver-k4r2h\" (UID: \"a0c7410f-98e2-4331-b24b-6eeab7f99401\") " pod="openshift-dns/node-resolver-k4r2h"
Jul 02 21:18:50.868756 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868088    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn6kr\" (UniqueName: \"kubernetes.io/projected/3dbcf436-a3ce-467a-b081-2cfbc20d3af5-kube-api-access-zn6kr\") pod \"iptables-alerter-qkjh4\" (UID: \"3dbcf436-a3ce-467a-b081-2cfbc20d3af5\") " pod="openshift-network-operator/iptables-alerter-qkjh4"
Jul 02 21:18:50.868756 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868111    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-cnibin\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.868756 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868147    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/240069b9-1019-4c78-9b12-e2a30f6baeb6-agent-certs\") pod \"konnectivity-agent-dnrb6\" (UID: \"240069b9-1019-4c78-9b12-e2a30f6baeb6\") " pod="kube-system/konnectivity-agent-dnrb6"
Jul 02 21:18:50.868756 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868186    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf253477-5a39-4648-9b18-4282286ab281-ovnkube-script-lib\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.868756 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868211    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b7e06c9-44c9-486b-8483-d71693ccc112-system-cni-dir\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.868756 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868276    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret\") pod \"global-pull-secret-syncer-gpccf\" (UID: \"3a6e6de1-8b1b-478e-8dce-996e81502c1e\") " pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:18:50.868756 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868307    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn2d7\" (UniqueName: \"kubernetes.io/projected/dda210c5-c607-473b-b4c9-7d97094ef2dd-kube-api-access-xn2d7\") pod \"aws-ebs-csi-driver-node-9rr6n\" (UID: \"dda210c5-c607-473b-b4c9-7d97094ef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:50.868756 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868331    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-host-kubelet\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.869619 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868373    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c7410f-98e2-4331-b24b-6eeab7f99401-tmp-dir\") pod \"node-resolver-k4r2h\" (UID: \"a0c7410f-98e2-4331-b24b-6eeab7f99401\") " pod="openshift-dns/node-resolver-k4r2h"
Jul 02 21:18:50.869619 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868451    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-system-cni-dir\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.869619 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868507    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-system-cni-dir\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.869619 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868539    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-host-run-multus-certs\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.869619 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868575    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-etc-kubernetes\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.869619 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868600    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-etc-openvswitch\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.869619 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868640    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c7410f-98e2-4331-b24b-6eeab7f99401-tmp-dir\") pod \"node-resolver-k4r2h\" (UID: \"a0c7410f-98e2-4331-b24b-6eeab7f99401\") " pod="openshift-dns/node-resolver-k4r2h"
Jul 02 21:18:50.869619 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868646    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-etc-kubernetes\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.869619 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868685    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-host-cni-bin\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.869619 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868708    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-host-run-multus-certs\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.869619 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868737    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn5c4\" (UniqueName: \"kubernetes.io/projected/bf253477-5a39-4648-9b18-4282286ab281-kube-api-access-bn5c4\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.869619 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868762    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-multus-conf-dir\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.869619 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868809    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a6e6de1-8b1b-478e-8dce-996e81502c1e-kubelet-config\") pod \"global-pull-secret-syncer-gpccf\" (UID: \"3a6e6de1-8b1b-478e-8dce-996e81502c1e\") " pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:18:50.869619 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868834    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs\") pod \"network-metrics-daemon-pxcm5\" (UID: \"28a7f381-3db6-43e5-9690-d409f048dcf2\") " pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:18:50.869619 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868858    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-host\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.869619 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868903    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a6e6de1-8b1b-478e-8dce-996e81502c1e-dbus\") pod \"global-pull-secret-syncer-gpccf\" (UID: \"3a6e6de1-8b1b-478e-8dce-996e81502c1e\") " pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:18:50.869619 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868960    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-multus-conf-dir\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.870445 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.868976    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dda210c5-c607-473b-b4c9-7d97094ef2dd-registration-dir\") pod \"aws-ebs-csi-driver-node-9rr6n\" (UID: \"dda210c5-c607-473b-b4c9-7d97094ef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:50.870445 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869006    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-host-cni-netd\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.870445 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869050    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf253477-5a39-4648-9b18-4282286ab281-ovn-node-metrics-cert\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.870445 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869084    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-etc-sysctl-conf\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.870445 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869106    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sqxf\" (UniqueName: \"kubernetes.io/projected/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-kube-api-access-2sqxf\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.870445 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869133    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-host-var-lib-kubelet\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.870445 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869163    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dda210c5-c607-473b-b4c9-7d97094ef2dd-device-dir\") pod \"aws-ebs-csi-driver-node-9rr6n\" (UID: \"dda210c5-c607-473b-b4c9-7d97094ef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:50.870445 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869189    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-host-var-lib-kubelet\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.870445 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869207    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dda210c5-c607-473b-b4c9-7d97094ef2dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-9rr6n\" (UID: \"dda210c5-c607-473b-b4c9-7d97094ef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:50.870445 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869255    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b7e06c9-44c9-486b-8483-d71693ccc112-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.870445 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869283    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-os-release\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.870445 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869308    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-host-run-k8s-cni-cncf-io\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.870445 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869332    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-host-var-lib-cni-multus\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.870445 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869353    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-hostroot\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.870445 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869375    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dda210c5-c607-473b-b4c9-7d97094ef2dd-socket-dir\") pod \"aws-ebs-csi-driver-node-9rr6n\" (UID: \"dda210c5-c607-473b-b4c9-7d97094ef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:50.870445 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869396    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-host-var-lib-cni-multus\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.870445 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869401    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/240069b9-1019-4c78-9b12-e2a30f6baeb6-konnectivity-ca\") pod \"konnectivity-agent-dnrb6\" (UID: \"240069b9-1019-4c78-9b12-e2a30f6baeb6\") " pod="kube-system/konnectivity-agent-dnrb6"
Jul 02 21:18:50.871303 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869416    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-host-run-k8s-cni-cncf-io\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.871303 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869426    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-os-release\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.871303 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869443    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-hostroot\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.871303 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869448    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-host-slash\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.871303 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869474    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-log-socket\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.871303 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869513    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dc228c55-194a-429b-a1dd-75001e9b6750-multus-daemon-config\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.871303 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869548    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83c0ff63-726d-4bb4-b4f6-0810ef1af2b2-host\") pod \"node-ca-9zmmn\" (UID: \"83c0ff63-726d-4bb4-b4f6-0810ef1af2b2\") " pod="openshift-image-registry/node-ca-9zmmn"
Jul 02 21:18:50.871303 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869596    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-974tj\" (UniqueName: \"kubernetes.io/projected/28a7f381-3db6-43e5-9690-d409f048dcf2-kube-api-access-974tj\") pod \"network-metrics-daemon-pxcm5\" (UID: \"28a7f381-3db6-43e5-9690-d409f048dcf2\") " pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:18:50.871303 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869644    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-etc-kubernetes\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.871303 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869666    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-sys\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.871303 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869688    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-lib-modules\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.871303 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869710    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-etc-tuned\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.871303 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869761    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-multus-socket-dir-parent\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.871303 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869785    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-etc-sysctl-d\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.871303 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869808    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-etc-systemd\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.871303 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869829    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-tmp\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.871303 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.869873    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dc228c55-194a-429b-a1dd-75001e9b6750-multus-socket-dir-parent\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.872085 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.870035    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dc228c55-194a-429b-a1dd-75001e9b6750-multus-daemon-config\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.877227 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.877190    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 02 21:18:50.881702 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.881628    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84h8t\" (UniqueName: \"kubernetes.io/projected/dc228c55-194a-429b-a1dd-75001e9b6750-kube-api-access-84h8t\") pod \"multus-w9fcp\" (UID: \"dc228c55-194a-429b-a1dd-75001e9b6750\") " pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:50.881702 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.881695    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vdcg\" (UniqueName: \"kubernetes.io/projected/a0c7410f-98e2-4331-b24b-6eeab7f99401-kube-api-access-5vdcg\") pod \"node-resolver-k4r2h\" (UID: \"a0c7410f-98e2-4331-b24b-6eeab7f99401\") " pod="openshift-dns/node-resolver-k4r2h"
Jul 02 21:18:50.882097 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.882076    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn6kr\" (UniqueName: \"kubernetes.io/projected/3dbcf436-a3ce-467a-b081-2cfbc20d3af5-kube-api-access-zn6kr\") pod \"iptables-alerter-qkjh4\" (UID: \"3dbcf436-a3ce-467a-b081-2cfbc20d3af5\") " pod="openshift-network-operator/iptables-alerter-qkjh4"
Jul 02 21:18:50.970374 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970341    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dda210c5-c607-473b-b4c9-7d97094ef2dd-registration-dir\") pod \"aws-ebs-csi-driver-node-9rr6n\" (UID: \"dda210c5-c607-473b-b4c9-7d97094ef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:50.970518 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970391    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-host-cni-netd\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.970518 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970410    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf253477-5a39-4648-9b18-4282286ab281-ovn-node-metrics-cert\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.970518 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970424    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-etc-sysctl-conf\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.970518 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970442    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sqxf\" (UniqueName: \"kubernetes.io/projected/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-kube-api-access-2sqxf\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.970518 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970441    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dda210c5-c607-473b-b4c9-7d97094ef2dd-registration-dir\") pod \"aws-ebs-csi-driver-node-9rr6n\" (UID: \"dda210c5-c607-473b-b4c9-7d97094ef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:50.970518 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970464    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dda210c5-c607-473b-b4c9-7d97094ef2dd-device-dir\") pod \"aws-ebs-csi-driver-node-9rr6n\" (UID: \"dda210c5-c607-473b-b4c9-7d97094ef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:50.970518 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970507    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-host-cni-netd\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.970518 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970520    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dda210c5-c607-473b-b4c9-7d97094ef2dd-device-dir\") pod \"aws-ebs-csi-driver-node-9rr6n\" (UID: \"dda210c5-c607-473b-b4c9-7d97094ef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:50.970914 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970539    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dda210c5-c607-473b-b4c9-7d97094ef2dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-9rr6n\" (UID: \"dda210c5-c607-473b-b4c9-7d97094ef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:50.970914 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970576    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b7e06c9-44c9-486b-8483-d71693ccc112-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.970914 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970607    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dda210c5-c607-473b-b4c9-7d97094ef2dd-socket-dir\") pod \"aws-ebs-csi-driver-node-9rr6n\" (UID: \"dda210c5-c607-473b-b4c9-7d97094ef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:50.970914 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970645    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-etc-sysctl-conf\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.970914 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970719    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dda210c5-c607-473b-b4c9-7d97094ef2dd-socket-dir\") pod \"aws-ebs-csi-driver-node-9rr6n\" (UID: \"dda210c5-c607-473b-b4c9-7d97094ef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:50.970914 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970712    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dda210c5-c607-473b-b4c9-7d97094ef2dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-9rr6n\" (UID: \"dda210c5-c607-473b-b4c9-7d97094ef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:50.970914 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970757    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/240069b9-1019-4c78-9b12-e2a30f6baeb6-konnectivity-ca\") pod \"konnectivity-agent-dnrb6\" (UID: \"240069b9-1019-4c78-9b12-e2a30f6baeb6\") " pod="kube-system/konnectivity-agent-dnrb6"
Jul 02 21:18:50.970914 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970799    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-host-slash\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.970914 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970820    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-log-socket\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.970914 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970846    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83c0ff63-726d-4bb4-b4f6-0810ef1af2b2-host\") pod \"node-ca-9zmmn\" (UID: \"83c0ff63-726d-4bb4-b4f6-0810ef1af2b2\") " pod="openshift-image-registry/node-ca-9zmmn"
Jul 02 21:18:50.970914 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970869    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-host-slash\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.970914 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970872    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-974tj\" (UniqueName: \"kubernetes.io/projected/28a7f381-3db6-43e5-9690-d409f048dcf2-kube-api-access-974tj\") pod \"network-metrics-daemon-pxcm5\" (UID: \"28a7f381-3db6-43e5-9690-d409f048dcf2\") " pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:18:50.970914 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970897    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-log-socket\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.970914 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970918    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-etc-kubernetes\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.971600 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970942    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-sys\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.971600 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970953    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83c0ff63-726d-4bb4-b4f6-0810ef1af2b2-host\") pod \"node-ca-9zmmn\" (UID: \"83c0ff63-726d-4bb4-b4f6-0810ef1af2b2\") " pod="openshift-image-registry/node-ca-9zmmn"
Jul 02 21:18:50.971600 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970963    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-lib-modules\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.971600 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.970985    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-etc-tuned\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.971600 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971001    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-etc-kubernetes\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.971600 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971007    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-etc-sysctl-d\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.971600 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971115    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-lib-modules\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.971600 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971121    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-etc-sysctl-d\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.971600 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971145    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-etc-systemd\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.971600 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971190    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-sys\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.971600 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971190    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-tmp\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.971600 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971203    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b7e06c9-44c9-486b-8483-d71693ccc112-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.971600 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971230    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-host-run-ovn-kubernetes\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.971600 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971262    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dda210c5-c607-473b-b4c9-7d97094ef2dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9rr6n\" (UID: \"dda210c5-c607-473b-b4c9-7d97094ef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:50.971600 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971266    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-etc-systemd\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.971600 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971300    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-host-run-ovn-kubernetes\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.971600 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971310    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dda210c5-c607-473b-b4c9-7d97094ef2dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9rr6n\" (UID: \"dda210c5-c607-473b-b4c9-7d97094ef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:50.971600 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971331    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-run-ovn\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.972440 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971357    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/83c0ff63-726d-4bb4-b4f6-0810ef1af2b2-serviceca\") pod \"node-ca-9zmmn\" (UID: \"83c0ff63-726d-4bb4-b4f6-0810ef1af2b2\") " pod="openshift-image-registry/node-ca-9zmmn"
Jul 02 21:18:50.972440 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971379    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-host-run-netns\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.972440 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971390    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/240069b9-1019-4c78-9b12-e2a30f6baeb6-konnectivity-ca\") pod \"konnectivity-agent-dnrb6\" (UID: \"240069b9-1019-4c78-9b12-e2a30f6baeb6\") " pod="kube-system/konnectivity-agent-dnrb6"
Jul 02 21:18:50.972440 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971404    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-var-lib-openvswitch\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.972440 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971413    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-run-ovn\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.972440 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971438    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-run-systemd\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.972440 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971461    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf253477-5a39-4648-9b18-4282286ab281-env-overrides\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.972440 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971462    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-host-run-netns\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.972440 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971482    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-var-lib-kubelet\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.972440 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971507    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vg5d\" (UniqueName: \"kubernetes.io/projected/4b7e06c9-44c9-486b-8483-d71693ccc112-kube-api-access-4vg5d\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.972440 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971538    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4b7e06c9-44c9-486b-8483-d71693ccc112-os-release\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.972440 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971561    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-systemd-units\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.972440 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971585    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-node-log\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.972440 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971608    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf253477-5a39-4648-9b18-4282286ab281-ovnkube-config\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.972440 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971635    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-etc-sysconfig\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.972440 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971662    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-run\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.972440 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971688    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b7e06c9-44c9-486b-8483-d71693ccc112-cni-binary-copy\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.973233 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971712    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b7e06c9-44c9-486b-8483-d71693ccc112-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.973233 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971738    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dda210c5-c607-473b-b4c9-7d97094ef2dd-sys-fs\") pod \"aws-ebs-csi-driver-node-9rr6n\" (UID: \"dda210c5-c607-473b-b4c9-7d97094ef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:50.973233 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971761    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxmd4\" (UniqueName: \"kubernetes.io/projected/83c0ff63-726d-4bb4-b4f6-0810ef1af2b2-kube-api-access-pxmd4\") pod \"node-ca-9zmmn\" (UID: \"83c0ff63-726d-4bb4-b4f6-0810ef1af2b2\") " pod="openshift-image-registry/node-ca-9zmmn"
Jul 02 21:18:50.973233 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971783    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-run-openvswitch\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.973233 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971792    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/83c0ff63-726d-4bb4-b4f6-0810ef1af2b2-serviceca\") pod \"node-ca-9zmmn\" (UID: \"83c0ff63-726d-4bb4-b4f6-0810ef1af2b2\") " pod="openshift-image-registry/node-ca-9zmmn"
Jul 02 21:18:50.973233 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971808    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.973233 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971846    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjksx\" (UniqueName: \"kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx\") pod \"network-check-target-b9tf7\" (UID: \"4076e581-5038-4786-9565-c72be1618034\") " pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:18:50.973233 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971855    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.973233 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971510    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-run-systemd\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.973233 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971877    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b7e06c9-44c9-486b-8483-d71693ccc112-cnibin\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.973233 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971904    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-etc-modprobe-d\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.973233 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971928    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b7e06c9-44c9-486b-8483-d71693ccc112-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.973233 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971930    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4b7e06c9-44c9-486b-8483-d71693ccc112-os-release\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.973233 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971538    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-var-lib-openvswitch\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.973233 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971950    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf253477-5a39-4648-9b18-4282286ab281-env-overrides\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.973233 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971958    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/240069b9-1019-4c78-9b12-e2a30f6baeb6-agent-certs\") pod \"konnectivity-agent-dnrb6\" (UID: \"240069b9-1019-4c78-9b12-e2a30f6baeb6\") " pod="kube-system/konnectivity-agent-dnrb6"
Jul 02 21:18:50.973233 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.971983    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf253477-5a39-4648-9b18-4282286ab281-ovnkube-script-lib\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.974022 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972006    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b7e06c9-44c9-486b-8483-d71693ccc112-system-cni-dir\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.974022 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972018    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-var-lib-kubelet\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.974022 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972030    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret\") pod \"global-pull-secret-syncer-gpccf\" (UID: \"3a6e6de1-8b1b-478e-8dce-996e81502c1e\") " pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:18:50.974022 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972058    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn2d7\" (UniqueName: \"kubernetes.io/projected/dda210c5-c607-473b-b4c9-7d97094ef2dd-kube-api-access-xn2d7\") pod \"aws-ebs-csi-driver-node-9rr6n\" (UID: \"dda210c5-c607-473b-b4c9-7d97094ef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:50.974022 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972072    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-run\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.974022 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972081    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-host-kubelet\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.974022 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972111    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-etc-openvswitch\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.974022 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972142    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-node-log\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.974022 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972147    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-host-cni-bin\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.974022 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972190    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn5c4\" (UniqueName: \"kubernetes.io/projected/bf253477-5a39-4648-9b18-4282286ab281-kube-api-access-bn5c4\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.974022 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972227    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a6e6de1-8b1b-478e-8dce-996e81502c1e-kubelet-config\") pod \"global-pull-secret-syncer-gpccf\" (UID: \"3a6e6de1-8b1b-478e-8dce-996e81502c1e\") " pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:18:50.974022 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972251    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs\") pod \"network-metrics-daemon-pxcm5\" (UID: \"28a7f381-3db6-43e5-9690-d409f048dcf2\") " pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:18:50.974022 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972276    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-host\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.974022 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972300    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a6e6de1-8b1b-478e-8dce-996e81502c1e-dbus\") pod \"global-pull-secret-syncer-gpccf\" (UID: \"3a6e6de1-8b1b-478e-8dce-996e81502c1e\") " pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:18:50.974022 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972443    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a6e6de1-8b1b-478e-8dce-996e81502c1e-dbus\") pod \"global-pull-secret-syncer-gpccf\" (UID: \"3a6e6de1-8b1b-478e-8dce-996e81502c1e\") " pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:18:50.974022 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972600    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b7e06c9-44c9-486b-8483-d71693ccc112-cni-binary-copy\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.974022 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972620    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf253477-5a39-4648-9b18-4282286ab281-ovnkube-config\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.974837 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972111    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-systemd-units\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.974837 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972713    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-host-cni-bin\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.974837 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972731    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-etc-modprobe-d\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.974837 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972776    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-etc-openvswitch\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.974837 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972817    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-host-kubelet\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.974837 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972933    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a6e6de1-8b1b-478e-8dce-996e81502c1e-kubelet-config\") pod \"global-pull-secret-syncer-gpccf\" (UID: \"3a6e6de1-8b1b-478e-8dce-996e81502c1e\") " pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:18:50.974837 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.972961    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b7e06c9-44c9-486b-8483-d71693ccc112-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.974837 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:50.973026    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:50.974837 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.973046    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b7e06c9-44c9-486b-8483-d71693ccc112-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.974837 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:50.973116    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs podName:28a7f381-3db6-43e5-9690-d409f048dcf2 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:51.473082039 +0000 UTC m=+3.099583905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs") pod "network-metrics-daemon-pxcm5" (UID: "28a7f381-3db6-43e5-9690-d409f048dcf2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:50.974837 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.973115    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-etc-sysconfig\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.974837 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.973184    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-host\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.974837 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.973189    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dda210c5-c607-473b-b4c9-7d97094ef2dd-sys-fs\") pod \"aws-ebs-csi-driver-node-9rr6n\" (UID: \"dda210c5-c607-473b-b4c9-7d97094ef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:50.974837 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.973333    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf253477-5a39-4648-9b18-4282286ab281-run-openvswitch\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.974837 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.973378    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b7e06c9-44c9-486b-8483-d71693ccc112-cnibin\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.974837 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:50.973409    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:50.974837 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.973453    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b7e06c9-44c9-486b-8483-d71693ccc112-system-cni-dir\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:50.975657 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:50.973478    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret podName:3a6e6de1-8b1b-478e-8dce-996e81502c1e nodeName:}" failed. No retries permitted until 2026-07-02 21:18:51.473460139 +0000 UTC m=+3.099962004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret") pod "global-pull-secret-syncer-gpccf" (UID: "3a6e6de1-8b1b-478e-8dce-996e81502c1e") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:50.975657 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.973667    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-tmp\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.975657 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.973745    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf253477-5a39-4648-9b18-4282286ab281-ovnkube-script-lib\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.975657 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.973954    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf253477-5a39-4648-9b18-4282286ab281-ovn-node-metrics-cert\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:50.975657 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.974280    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-etc-tuned\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:50.975657 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.975546    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/240069b9-1019-4c78-9b12-e2a30f6baeb6-agent-certs\") pod \"konnectivity-agent-dnrb6\" (UID: \"240069b9-1019-4c78-9b12-e2a30f6baeb6\") " pod="kube-system/konnectivity-agent-dnrb6"
Jul 02 21:18:50.991983 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:50.991934    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sqxf\" (UniqueName: \"kubernetes.io/projected/3ba76650-ff51-4f61-bbe0-cee05f1b91c6-kube-api-access-2sqxf\") pod \"tuned-bb2r6\" (UID: \"3ba76650-ff51-4f61-bbe0-cee05f1b91c6\") " pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:51.004845 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:51.004824    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxmd4\" (UniqueName: \"kubernetes.io/projected/83c0ff63-726d-4bb4-b4f6-0810ef1af2b2-kube-api-access-pxmd4\") pod \"node-ca-9zmmn\" (UID: \"83c0ff63-726d-4bb4-b4f6-0810ef1af2b2\") " pod="openshift-image-registry/node-ca-9zmmn"
Jul 02 21:18:51.006022 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:51.005985    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vg5d\" (UniqueName: \"kubernetes.io/projected/4b7e06c9-44c9-486b-8483-d71693ccc112-kube-api-access-4vg5d\") pod \"multus-additional-cni-plugins-rfbk7\" (UID: \"4b7e06c9-44c9-486b-8483-d71693ccc112\") " pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:51.007106 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:51.007084    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn2d7\" (UniqueName: \"kubernetes.io/projected/dda210c5-c607-473b-b4c9-7d97094ef2dd-kube-api-access-xn2d7\") pod \"aws-ebs-csi-driver-node-9rr6n\" (UID: \"dda210c5-c607-473b-b4c9-7d97094ef2dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:51.007548 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:51.007529    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:18:51.007642 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:51.007552    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:18:51.007642 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:51.007565    2575 projected.go:196] Error preparing data for projected volume kube-api-access-qjksx for pod openshift-network-diagnostics/network-check-target-b9tf7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:51.007642 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:51.007639    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx podName:4076e581-5038-4786-9565-c72be1618034 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:51.507623006 +0000 UTC m=+3.134124873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qjksx" (UniqueName: "kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx") pod "network-check-target-b9tf7" (UID: "4076e581-5038-4786-9565-c72be1618034") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:51.008676 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:51.008657    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-974tj\" (UniqueName: \"kubernetes.io/projected/28a7f381-3db6-43e5-9690-d409f048dcf2-kube-api-access-974tj\") pod \"network-metrics-daemon-pxcm5\" (UID: \"28a7f381-3db6-43e5-9690-d409f048dcf2\") " pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:18:51.008807 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:51.008783    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn5c4\" (UniqueName: \"kubernetes.io/projected/bf253477-5a39-4648-9b18-4282286ab281-kube-api-access-bn5c4\") pod \"ovnkube-node-2725k\" (UID: \"bf253477-5a39-4648-9b18-4282286ab281\") " pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:51.063768 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:51.063741    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k4r2h"
Jul 02 21:18:51.073001 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:51.072982    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w9fcp"
Jul 02 21:18:51.082624 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:51.082602    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qkjh4"
Jul 02 21:18:51.089216 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:51.089199    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dnrb6"
Jul 02 21:18:51.096836 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:51.096820    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bb2r6"
Jul 02 21:18:51.103489 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:51.103471    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:18:51.112978 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:51.112960    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rfbk7"
Jul 02 21:18:51.118485 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:51.118466    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n"
Jul 02 21:18:51.125066 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:51.125042    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9zmmn"
Jul 02 21:18:51.295354 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:51.295279    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:18:51.476038 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:51.476006    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret\") pod \"global-pull-secret-syncer-gpccf\" (UID: \"3a6e6de1-8b1b-478e-8dce-996e81502c1e\") " pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:18:51.476038 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:51.476045    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs\") pod \"network-metrics-daemon-pxcm5\" (UID: \"28a7f381-3db6-43e5-9690-d409f048dcf2\") " pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:18:51.476272 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:51.476161    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:51.476272 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:51.476241    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret podName:3a6e6de1-8b1b-478e-8dce-996e81502c1e nodeName:}" failed. No retries permitted until 2026-07-02 21:18:52.476223226 +0000 UTC m=+4.102725103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret") pod "global-pull-secret-syncer-gpccf" (UID: "3a6e6de1-8b1b-478e-8dce-996e81502c1e") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:51.476272 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:51.476161    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:51.476444 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:51.476341    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs podName:28a7f381-3db6-43e5-9690-d409f048dcf2 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:52.476319331 +0000 UTC m=+4.102821203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs") pod "network-metrics-daemon-pxcm5" (UID: "28a7f381-3db6-43e5-9690-d409f048dcf2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:51.576597 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:51.576526    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjksx\" (UniqueName: \"kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx\") pod \"network-check-target-b9tf7\" (UID: \"4076e581-5038-4786-9565-c72be1618034\") " pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:18:51.576735 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:51.576635    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:18:51.576735 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:51.576648    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:18:51.576735 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:51.576656    2575 projected.go:196] Error preparing data for projected volume kube-api-access-qjksx for pod openshift-network-diagnostics/network-check-target-b9tf7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:51.576735 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:51.576699    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx podName:4076e581-5038-4786-9565-c72be1618034 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:52.576686832 +0000 UTC m=+4.203188686 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qjksx" (UniqueName: "kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx") pod "network-check-target-b9tf7" (UID: "4076e581-5038-4786-9565-c72be1618034") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:51.804612 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:51.804576    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 21:13:50 +0000 UTC" deadline="2028-02-09 03:29:01.442463346 +0000 UTC"
Jul 02 21:18:51.804612 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:51.804604    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14070h10m9.637862875s"
Jul 02 21:18:51.956619 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:51.956579    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240069b9_1019_4c78_9b12_e2a30f6baeb6.slice/crio-f3e06b0ea72d806e0c3dcf65ca886c659a4dc7291a8bdddb22c3fd25087668ad WatchSource:0}: Error finding container f3e06b0ea72d806e0c3dcf65ca886c659a4dc7291a8bdddb22c3fd25087668ad: Status 404 returned error can't find the container with id f3e06b0ea72d806e0c3dcf65ca886c659a4dc7291a8bdddb22c3fd25087668ad
Jul 02 21:18:51.959064 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:51.959030    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c0ff63_726d_4bb4_b4f6_0810ef1af2b2.slice/crio-668f1976ab6b74d497f854470ffef3a9ecd646ad8fa3d2e97e13c2512ceeb8c9 WatchSource:0}: Error finding container 668f1976ab6b74d497f854470ffef3a9ecd646ad8fa3d2e97e13c2512ceeb8c9: Status 404 returned error can't find the container with id 668f1976ab6b74d497f854470ffef3a9ecd646ad8fa3d2e97e13c2512ceeb8c9
Jul 02 21:18:51.961615 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:51.961590    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba76650_ff51_4f61_bbe0_cee05f1b91c6.slice/crio-be7e08cb1bd827317960539605a00b59a5356a11cb07e95a0fdddf3f6db80309 WatchSource:0}: Error finding container be7e08cb1bd827317960539605a00b59a5356a11cb07e95a0fdddf3f6db80309: Status 404 returned error can't find the container with id be7e08cb1bd827317960539605a00b59a5356a11cb07e95a0fdddf3f6db80309
Jul 02 21:18:51.962723 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:51.962583    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbcf436_a3ce_467a_b081_2cfbc20d3af5.slice/crio-a825eecf46bba535985437ac4a4816c642e36b6f1a3b320ba14411e203546b1e WatchSource:0}: Error finding container a825eecf46bba535985437ac4a4816c642e36b6f1a3b320ba14411e203546b1e: Status 404 returned error can't find the container with id a825eecf46bba535985437ac4a4816c642e36b6f1a3b320ba14411e203546b1e
Jul 02 21:18:51.963520 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:18:51.963499    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c7410f_98e2_4331_b24b_6eeab7f99401.slice/crio-e7f6cb1f14884ee7d2e6ddf28b0a9742f836c7d17c4d59f1aee24f54550683c7 WatchSource:0}: Error finding container e7f6cb1f14884ee7d2e6ddf28b0a9742f836c7d17c4d59f1aee24f54550683c7: Status 404 returned error can't find the container with id e7f6cb1f14884ee7d2e6ddf28b0a9742f836c7d17c4d59f1aee24f54550683c7
Jul 02 21:18:52.008799 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:52.008677    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:18:52.008894 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:52.008706    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:18:52.008894 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:52.008883    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxcm5" podUID="28a7f381-3db6-43e5-9690-d409f048dcf2"
Jul 02 21:18:52.008995 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:52.008976    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gpccf" podUID="3a6e6de1-8b1b-478e-8dce-996e81502c1e"
Jul 02 21:18:52.024686 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:52.024658    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w9fcp" event={"ID":"dc228c55-194a-429b-a1dd-75001e9b6750","Type":"ContainerStarted","Data":"cc3c1a966f4f956e5a896a37722698f4c695f075deb355c2ba3ac178fcbaae8f"}
Jul 02 21:18:52.025789 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:52.025767    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k4r2h" event={"ID":"a0c7410f-98e2-4331-b24b-6eeab7f99401","Type":"ContainerStarted","Data":"e7f6cb1f14884ee7d2e6ddf28b0a9742f836c7d17c4d59f1aee24f54550683c7"}
Jul 02 21:18:52.026955 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:52.026936    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9zmmn" event={"ID":"83c0ff63-726d-4bb4-b4f6-0810ef1af2b2","Type":"ContainerStarted","Data":"668f1976ab6b74d497f854470ffef3a9ecd646ad8fa3d2e97e13c2512ceeb8c9"}
Jul 02 21:18:52.027919 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:52.027895    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dnrb6" event={"ID":"240069b9-1019-4c78-9b12-e2a30f6baeb6","Type":"ContainerStarted","Data":"f3e06b0ea72d806e0c3dcf65ca886c659a4dc7291a8bdddb22c3fd25087668ad"}
Jul 02 21:18:52.029003 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:52.028979    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2725k" event={"ID":"bf253477-5a39-4648-9b18-4282286ab281","Type":"ContainerStarted","Data":"ad4b1db63031b239a84cbb0c90298ed209a7813d8d6db43c24703be37b8a49d8"}
Jul 02 21:18:52.030085 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:52.030059    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rfbk7" event={"ID":"4b7e06c9-44c9-486b-8483-d71693ccc112","Type":"ContainerStarted","Data":"3ac8db6945bc65c35e0f4fdac6cb11f82232684e01bf16bab46383c7ad954b4a"}
Jul 02 21:18:52.031139 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:52.031118    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n" event={"ID":"dda210c5-c607-473b-b4c9-7d97094ef2dd","Type":"ContainerStarted","Data":"14850d69b09af7d9a26dacd8ee397cdb507077467b5b8e7088ba6db62229fcb1"}
Jul 02 21:18:52.032073 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:52.032050    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qkjh4" event={"ID":"3dbcf436-a3ce-467a-b081-2cfbc20d3af5","Type":"ContainerStarted","Data":"a825eecf46bba535985437ac4a4816c642e36b6f1a3b320ba14411e203546b1e"}
Jul 02 21:18:52.033018 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:52.033001    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bb2r6" event={"ID":"3ba76650-ff51-4f61-bbe0-cee05f1b91c6","Type":"ContainerStarted","Data":"be7e08cb1bd827317960539605a00b59a5356a11cb07e95a0fdddf3f6db80309"}
Jul 02 21:18:52.482496 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:52.482421    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret\") pod \"global-pull-secret-syncer-gpccf\" (UID: \"3a6e6de1-8b1b-478e-8dce-996e81502c1e\") " pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:18:52.482496 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:52.482473    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs\") pod \"network-metrics-daemon-pxcm5\" (UID: \"28a7f381-3db6-43e5-9690-d409f048dcf2\") " pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:18:52.482760 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:52.482624    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:52.482760 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:52.482681    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs podName:28a7f381-3db6-43e5-9690-d409f048dcf2 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:54.482662099 +0000 UTC m=+6.109163956 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs") pod "network-metrics-daemon-pxcm5" (UID: "28a7f381-3db6-43e5-9690-d409f048dcf2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:52.483069 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:52.483050    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:52.483161 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:52.483104    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret podName:3a6e6de1-8b1b-478e-8dce-996e81502c1e nodeName:}" failed. No retries permitted until 2026-07-02 21:18:54.483090086 +0000 UTC m=+6.109591945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret") pod "global-pull-secret-syncer-gpccf" (UID: "3a6e6de1-8b1b-478e-8dce-996e81502c1e") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:52.583978 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:52.583811    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjksx\" (UniqueName: \"kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx\") pod \"network-check-target-b9tf7\" (UID: \"4076e581-5038-4786-9565-c72be1618034\") " pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:18:52.584132 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:52.583982    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:18:52.584132 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:52.584005    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:18:52.584132 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:52.584017    2575 projected.go:196] Error preparing data for projected volume kube-api-access-qjksx for pod openshift-network-diagnostics/network-check-target-b9tf7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:52.584132 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:52.584070    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx podName:4076e581-5038-4786-9565-c72be1618034 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:54.584052574 +0000 UTC m=+6.210554431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qjksx" (UniqueName: "kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx") pod "network-check-target-b9tf7" (UID: "4076e581-5038-4786-9565-c72be1618034") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:52.805823 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:52.805717    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 21:13:50 +0000 UTC" deadline="2028-03-31 09:14:25.682795554 +0000 UTC"
Jul 02 21:18:52.805823 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:52.805776    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15299h55m32.877023323s"
Jul 02 21:18:53.011774 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:53.011253    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:18:53.011774 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:53.011366    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9tf7" podUID="4076e581-5038-4786-9565-c72be1618034"
Jul 02 21:18:53.049197 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:53.048765    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-227.ec2.internal" event={"ID":"c947105a64a6be554217ac8fde87371e","Type":"ContainerStarted","Data":"4415751278caed45c9753d294fba22535d941cb3bcae2b6f68f2447cdb54f92a"}
Jul 02 21:18:53.067049 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:53.066327    2575 generic.go:356] "Generic (PLEG): container finished" podID="912e0ce5f47be8c71daf0c28291ccddf" containerID="1441e8667f003d2858565871f845bd0be19e8dab9b376cb335e2d2e70c574785" exitCode=0
Jul 02 21:18:53.067049 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:53.066376    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-227.ec2.internal" event={"ID":"912e0ce5f47be8c71daf0c28291ccddf","Type":"ContainerDied","Data":"1441e8667f003d2858565871f845bd0be19e8dab9b376cb335e2d2e70c574785"}
Jul 02 21:18:53.083632 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:53.083543    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-227.ec2.internal" podStartSLOduration=3.083524603 podStartE2EDuration="3.083524603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:18:53.066027935 +0000 UTC m=+4.692529812" watchObservedRunningTime="2026-07-02 21:18:53.083524603 +0000 UTC m=+4.710026481"
Jul 02 21:18:54.009750 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:54.009083    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:18:54.009750 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:54.009234    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxcm5" podUID="28a7f381-3db6-43e5-9690-d409f048dcf2"
Jul 02 21:18:54.009750 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:54.009591    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:18:54.009750 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:54.009691    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gpccf" podUID="3a6e6de1-8b1b-478e-8dce-996e81502c1e"
Jul 02 21:18:54.090802 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:54.090723    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-227.ec2.internal" event={"ID":"912e0ce5f47be8c71daf0c28291ccddf","Type":"ContainerStarted","Data":"a7462c5dab6db04fc39d2e6c8170c75200be6c3574da445ee7a3d303567cb50d"}
Jul 02 21:18:54.503221 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:54.503119    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret\") pod \"global-pull-secret-syncer-gpccf\" (UID: \"3a6e6de1-8b1b-478e-8dce-996e81502c1e\") " pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:18:54.503221 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:54.503189    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs\") pod \"network-metrics-daemon-pxcm5\" (UID: \"28a7f381-3db6-43e5-9690-d409f048dcf2\") " pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:18:54.503486 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:54.503308    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:54.503486 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:54.503400    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs podName:28a7f381-3db6-43e5-9690-d409f048dcf2 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:58.503381965 +0000 UTC m=+10.129883833 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs") pod "network-metrics-daemon-pxcm5" (UID: "28a7f381-3db6-43e5-9690-d409f048dcf2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:54.503817 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:54.503795    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:54.503908 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:54.503848    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret podName:3a6e6de1-8b1b-478e-8dce-996e81502c1e nodeName:}" failed. No retries permitted until 2026-07-02 21:18:58.503834602 +0000 UTC m=+10.130336479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret") pod "global-pull-secret-syncer-gpccf" (UID: "3a6e6de1-8b1b-478e-8dce-996e81502c1e") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:54.603838 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:54.603785    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjksx\" (UniqueName: \"kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx\") pod \"network-check-target-b9tf7\" (UID: \"4076e581-5038-4786-9565-c72be1618034\") " pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:18:54.603989 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:54.603926    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:18:54.603989 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:54.603950    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:18:54.603989 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:54.603962    2575 projected.go:196] Error preparing data for projected volume kube-api-access-qjksx for pod openshift-network-diagnostics/network-check-target-b9tf7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:54.604145 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:54.604020    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx podName:4076e581-5038-4786-9565-c72be1618034 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:58.604002838 +0000 UTC m=+10.230504695 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qjksx" (UniqueName: "kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx") pod "network-check-target-b9tf7" (UID: "4076e581-5038-4786-9565-c72be1618034") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:55.008923 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:55.008894    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:18:55.009081 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:55.009019    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9tf7" podUID="4076e581-5038-4786-9565-c72be1618034"
Jul 02 21:18:56.008828 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:56.008768    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:18:56.009285 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:56.008893    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gpccf" podUID="3a6e6de1-8b1b-478e-8dce-996e81502c1e"
Jul 02 21:18:56.009285 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:56.009278    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:18:56.009430 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:56.009392    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxcm5" podUID="28a7f381-3db6-43e5-9690-d409f048dcf2"
Jul 02 21:18:57.009250 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:57.009163    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:18:57.009697 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:57.009310    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9tf7" podUID="4076e581-5038-4786-9565-c72be1618034"
Jul 02 21:18:58.009394 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:58.009310    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:18:58.009918 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:58.009434    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gpccf" podUID="3a6e6de1-8b1b-478e-8dce-996e81502c1e"
Jul 02 21:18:58.009918 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:58.009500    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:18:58.009918 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:58.009634    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxcm5" podUID="28a7f381-3db6-43e5-9690-d409f048dcf2"
Jul 02 21:18:58.538381 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:58.538228    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret\") pod \"global-pull-secret-syncer-gpccf\" (UID: \"3a6e6de1-8b1b-478e-8dce-996e81502c1e\") " pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:18:58.538381 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:58.538291    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs\") pod \"network-metrics-daemon-pxcm5\" (UID: \"28a7f381-3db6-43e5-9690-d409f048dcf2\") " pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:18:58.538637 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:58.538396    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:58.538637 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:58.538452    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:58.538637 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:58.538475    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret podName:3a6e6de1-8b1b-478e-8dce-996e81502c1e nodeName:}" failed. No retries permitted until 2026-07-02 21:19:06.538456626 +0000 UTC m=+18.164958486 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret") pod "global-pull-secret-syncer-gpccf" (UID: "3a6e6de1-8b1b-478e-8dce-996e81502c1e") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:58.538637 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:58.538516    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs podName:28a7f381-3db6-43e5-9690-d409f048dcf2 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:06.538495446 +0000 UTC m=+18.164997314 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs") pod "network-metrics-daemon-pxcm5" (UID: "28a7f381-3db6-43e5-9690-d409f048dcf2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:58.639332 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:58.639287    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjksx\" (UniqueName: \"kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx\") pod \"network-check-target-b9tf7\" (UID: \"4076e581-5038-4786-9565-c72be1618034\") " pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:18:58.639560 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:58.639536    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:18:58.639647 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:58.639568    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:18:58.639647 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:58.639585    2575 projected.go:196] Error preparing data for projected volume kube-api-access-qjksx for pod openshift-network-diagnostics/network-check-target-b9tf7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:58.639748 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:58.639649    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx podName:4076e581-5038-4786-9565-c72be1618034 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:06.639631054 +0000 UTC m=+18.266132911 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qjksx" (UniqueName: "kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx") pod "network-check-target-b9tf7" (UID: "4076e581-5038-4786-9565-c72be1618034") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:59.010595 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:18:59.010315    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:18:59.010595 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:18:59.010424    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9tf7" podUID="4076e581-5038-4786-9565-c72be1618034"
Jul 02 21:19:00.009096 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:00.008814    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:19:00.009096 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:00.008856    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:19:00.009096 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:00.008957    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gpccf" podUID="3a6e6de1-8b1b-478e-8dce-996e81502c1e"
Jul 02 21:19:00.009096 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:00.009062    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxcm5" podUID="28a7f381-3db6-43e5-9690-d409f048dcf2"
Jul 02 21:19:01.010125 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:01.010085    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:19:01.010573 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:01.010245    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9tf7" podUID="4076e581-5038-4786-9565-c72be1618034"
Jul 02 21:19:02.009106 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:02.009075    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:19:02.009313 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:02.009079    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:19:02.009313 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:02.009222    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gpccf" podUID="3a6e6de1-8b1b-478e-8dce-996e81502c1e"
Jul 02 21:19:02.009313 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:02.009287    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxcm5" podUID="28a7f381-3db6-43e5-9690-d409f048dcf2"
Jul 02 21:19:03.008899 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:03.008822    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:19:03.009390 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:03.008950    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9tf7" podUID="4076e581-5038-4786-9565-c72be1618034"
Jul 02 21:19:04.009623 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:04.009583    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:19:04.010028 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:04.009638    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:19:04.010028 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:04.009720    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gpccf" podUID="3a6e6de1-8b1b-478e-8dce-996e81502c1e"
Jul 02 21:19:04.010028 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:04.009859    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxcm5" podUID="28a7f381-3db6-43e5-9690-d409f048dcf2"
Jul 02 21:19:05.008884 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:05.008841    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:19:05.009046 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:05.008975    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9tf7" podUID="4076e581-5038-4786-9565-c72be1618034"
Jul 02 21:19:06.008571 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:06.008539    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:19:06.009038 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:06.008666    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gpccf" podUID="3a6e6de1-8b1b-478e-8dce-996e81502c1e"
Jul 02 21:19:06.009038 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:06.008698    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:19:06.009038 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:06.008841    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxcm5" podUID="28a7f381-3db6-43e5-9690-d409f048dcf2"
Jul 02 21:19:06.603500 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:06.603458    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret\") pod \"global-pull-secret-syncer-gpccf\" (UID: \"3a6e6de1-8b1b-478e-8dce-996e81502c1e\") " pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:19:06.603688 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:06.603515    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs\") pod \"network-metrics-daemon-pxcm5\" (UID: \"28a7f381-3db6-43e5-9690-d409f048dcf2\") " pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:19:06.603688 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:06.603625    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:06.603688 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:06.603625    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:06.603688 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:06.603687    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs podName:28a7f381-3db6-43e5-9690-d409f048dcf2 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:22.603666951 +0000 UTC m=+34.230168808 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs") pod "network-metrics-daemon-pxcm5" (UID: "28a7f381-3db6-43e5-9690-d409f048dcf2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:06.603904 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:06.603705    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret podName:3a6e6de1-8b1b-478e-8dce-996e81502c1e nodeName:}" failed. No retries permitted until 2026-07-02 21:19:22.603695476 +0000 UTC m=+34.230197339 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret") pod "global-pull-secret-syncer-gpccf" (UID: "3a6e6de1-8b1b-478e-8dce-996e81502c1e") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:06.704839 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:06.704797    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjksx\" (UniqueName: \"kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx\") pod \"network-check-target-b9tf7\" (UID: \"4076e581-5038-4786-9565-c72be1618034\") " pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:19:06.705030 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:06.704956    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:19:06.705030 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:06.704983    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:19:06.705030 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:06.704993    2575 projected.go:196] Error preparing data for projected volume kube-api-access-qjksx for pod openshift-network-diagnostics/network-check-target-b9tf7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:06.705208 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:06.705043    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx podName:4076e581-5038-4786-9565-c72be1618034 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:22.705030458 +0000 UTC m=+34.331532313 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qjksx" (UniqueName: "kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx") pod "network-check-target-b9tf7" (UID: "4076e581-5038-4786-9565-c72be1618034") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:07.009085 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:07.009004    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:19:07.009537 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:07.009134    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9tf7" podUID="4076e581-5038-4786-9565-c72be1618034"
Jul 02 21:19:08.009059 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:08.009020    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:19:08.009244 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:08.009034    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:19:08.009244 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:08.009187    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxcm5" podUID="28a7f381-3db6-43e5-9690-d409f048dcf2"
Jul 02 21:19:08.009582 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:08.009253    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gpccf" podUID="3a6e6de1-8b1b-478e-8dce-996e81502c1e"
Jul 02 21:19:09.009652 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:09.009615    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:19:09.010040 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:09.009709    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9tf7" podUID="4076e581-5038-4786-9565-c72be1618034"
Jul 02 21:19:10.009236 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:10.009202    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:19:10.009437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:10.009208    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:19:10.009437 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:10.009338    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gpccf" podUID="3a6e6de1-8b1b-478e-8dce-996e81502c1e"
Jul 02 21:19:10.009555 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:10.009436    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxcm5" podUID="28a7f381-3db6-43e5-9690-d409f048dcf2"
Jul 02 21:19:11.009681 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.009334    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:19:11.010415 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:11.009822    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9tf7" podUID="4076e581-5038-4786-9565-c72be1618034"
Jul 02 21:19:11.125983 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.125786    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2725k" event={"ID":"bf253477-5a39-4648-9b18-4282286ab281","Type":"ContainerStarted","Data":"12506e834d9d507a0d0ec827a77cf766081478a6c2fc968d7056acc45f49cefc"}
Jul 02 21:19:11.125983 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.125898    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2725k" event={"ID":"bf253477-5a39-4648-9b18-4282286ab281","Type":"ContainerStarted","Data":"862c6a53e34b615146f6f4805e2c60bb60115d7fdef90e7b50b9902f6a5cf6ba"}
Jul 02 21:19:11.125983 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.125959    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2725k" event={"ID":"bf253477-5a39-4648-9b18-4282286ab281","Type":"ContainerStarted","Data":"7400e84592b2788371cbf7217d86f34d06f3b3daf640647c1294760f09b11aa4"}
Jul 02 21:19:11.126296 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.126011    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2725k" event={"ID":"bf253477-5a39-4648-9b18-4282286ab281","Type":"ContainerStarted","Data":"7d1cb6a210730b95198438e48d37765ce902fed24ad64553a1ff97002daa46b6"}
Jul 02 21:19:11.126296 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.126060    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2725k" event={"ID":"bf253477-5a39-4648-9b18-4282286ab281","Type":"ContainerStarted","Data":"0304bfb8684999bbebaab6287e42f7507ae85028f00598e9d973972189b3c63d"}
Jul 02 21:19:11.126296 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.126108    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2725k" event={"ID":"bf253477-5a39-4648-9b18-4282286ab281","Type":"ContainerStarted","Data":"18a35025ce9619d548c11325f67e793b69d7656e5bc8f431f366da3285624309"}
Jul 02 21:19:11.127591 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.127561    2575 generic.go:356] "Generic (PLEG): container finished" podID="4b7e06c9-44c9-486b-8483-d71693ccc112" containerID="602773f10ce1d14dbaad66f2d3d459e7161b065f1165e209aa2ee546103d4e65" exitCode=0
Jul 02 21:19:11.127731 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.127626    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rfbk7" event={"ID":"4b7e06c9-44c9-486b-8483-d71693ccc112","Type":"ContainerDied","Data":"602773f10ce1d14dbaad66f2d3d459e7161b065f1165e209aa2ee546103d4e65"}
Jul 02 21:19:11.129591 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.129568    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n" event={"ID":"dda210c5-c607-473b-b4c9-7d97094ef2dd","Type":"ContainerStarted","Data":"c3d6eddcd56978e6b7a9cd370f5da7ec79c7d62281e78b7bb12bb0f2211ca1ff"}
Jul 02 21:19:11.131381 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.131354    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bb2r6" event={"ID":"3ba76650-ff51-4f61-bbe0-cee05f1b91c6","Type":"ContainerStarted","Data":"d5b699a720bd14bf33f6ccec08cee2566646e3c2124adc64cec1464d28d8ab36"}
Jul 02 21:19:11.132946 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.132923    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w9fcp" event={"ID":"dc228c55-194a-429b-a1dd-75001e9b6750","Type":"ContainerStarted","Data":"1b6287b361e7f29f420dd8d9b838dd1ec749df23b6964e12df23c7f8ffb8e652"}
Jul 02 21:19:11.134485 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.134461    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k4r2h" event={"ID":"a0c7410f-98e2-4331-b24b-6eeab7f99401","Type":"ContainerStarted","Data":"75b777f44f101b6af3a644058719d4fd7da15a25afc9db9c30f7c1da9e467643"}
Jul 02 21:19:11.135990 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.135948    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9zmmn" event={"ID":"83c0ff63-726d-4bb4-b4f6-0810ef1af2b2","Type":"ContainerStarted","Data":"0d96d70a0b2f0c6b078c8b5f75f035114fe55695eb1280f6ac559155e6d2380f"}
Jul 02 21:19:11.137483 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.137460    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dnrb6" event={"ID":"240069b9-1019-4c78-9b12-e2a30f6baeb6","Type":"ContainerStarted","Data":"a06acd4bec86022d8ea828a2ba12f383221fb41d4f21d46c918b811d0dd460ba"}
Jul 02 21:19:11.159115 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.159069    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-227.ec2.internal" podStartSLOduration=21.159055054 podStartE2EDuration="21.159055054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:18:54.106803312 +0000 UTC m=+5.733305204" watchObservedRunningTime="2026-07-02 21:19:11.159055054 +0000 UTC m=+22.785556930"
Jul 02 21:19:11.178837 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.178765    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-w9fcp" podStartSLOduration=3.690320342 podStartE2EDuration="22.178746228s" totalImagesPullingTime="18.488425886s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:11.178542181 +0000 UTC m=+22.805044060" watchObservedRunningTime="2026-07-02 21:19:11.178746228 +0000 UTC m=+22.805248105"
Jul 02 21:19:11.194895 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.194779    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dnrb6" podStartSLOduration=11.731155736 podStartE2EDuration="22.194750282s" totalImagesPullingTime="10.463594546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:11.194568143 +0000 UTC m=+22.821070020" watchObservedRunningTime="2026-07-02 21:19:11.194750282 +0000 UTC m=+22.821252163"
Jul 02 21:19:11.213360 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.213305    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bb2r6" podStartSLOduration=3.77007264 podStartE2EDuration="22.213290206s" totalImagesPullingTime="18.443217566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:11.212747666 +0000 UTC m=+22.839249541" watchObservedRunningTime="2026-07-02 21:19:11.213290206 +0000 UTC m=+22.839792079"
Jul 02 21:19:11.252482 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.252425    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9zmmn" podStartSLOduration=3.803293623 podStartE2EDuration="22.252403732s" totalImagesPullingTime="18.449110109s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:11.230973344 +0000 UTC m=+22.857475220" watchObservedRunningTime="2026-07-02 21:19:11.252403732 +0000 UTC m=+22.878905608"
Jul 02 21:19:11.252879 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.252851    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-k4r2h" podStartSLOduration=3.814135832 podStartE2EDuration="22.252842067s" totalImagesPullingTime="18.438706235s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:11.252352843 +0000 UTC m=+22.878854719" watchObservedRunningTime="2026-07-02 21:19:11.252842067 +0000 UTC m=+22.879343942"
Jul 02 21:19:11.677598 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.677555    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 02 21:19:11.893502 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.893309    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T21:19:11.677577939Z","UUID":"e5701e04-f691-4985-80e7-f29c1f6c25b4","Handler":null,"Name":"","Endpoint":""}
Jul 02 21:19:11.894957 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.894931    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 02 21:19:11.895086 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:11.894976    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 02 21:19:12.009528 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:12.009496    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:19:12.009759 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:12.009496    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:19:12.009759 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:12.009597    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gpccf" podUID="3a6e6de1-8b1b-478e-8dce-996e81502c1e"
Jul 02 21:19:12.009759 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:12.009680    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxcm5" podUID="28a7f381-3db6-43e5-9690-d409f048dcf2"
Jul 02 21:19:12.142125 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:12.142088    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n" event={"ID":"dda210c5-c607-473b-b4c9-7d97094ef2dd","Type":"ContainerStarted","Data":"91e2d60b7935505a98f117f5f2197ee68c257c799a63c5fbfe93714279fd5cb8"}
Jul 02 21:19:12.144068 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:12.143966    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qkjh4" event={"ID":"3dbcf436-a3ce-467a-b081-2cfbc20d3af5","Type":"ContainerStarted","Data":"4bcaee3139f8d4e0d7956792ef54d4baeed57956d541c5befa668d5fcb28a609"}
Jul 02 21:19:12.162096 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:12.162054    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qkjh4" podStartSLOduration=4.720702301 podStartE2EDuration="23.162039878s" totalImagesPullingTime="18.441337577s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:12.161756209 +0000 UTC m=+23.788258088" watchObservedRunningTime="2026-07-02 21:19:12.162039878 +0000 UTC m=+23.788541754"
Jul 02 21:19:13.009342 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:13.009157    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:19:13.009508 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:13.009446    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9tf7" podUID="4076e581-5038-4786-9565-c72be1618034"
Jul 02 21:19:13.149048 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:13.148949    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n" event={"ID":"dda210c5-c607-473b-b4c9-7d97094ef2dd","Type":"ContainerStarted","Data":"d91d8cfef000e4e2859347ccf7e261497cc0dd5b47c14cac1b904eaa6e0e7ebe"}
Jul 02 21:19:13.171485 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:13.171432    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rr6n" podStartSLOduration=3.682116374 podStartE2EDuration="24.171413074s" totalImagesPullingTime="20.4892967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:13.170655539 +0000 UTC m=+24.797157416" watchObservedRunningTime="2026-07-02 21:19:13.171413074 +0000 UTC m=+24.797914951"
Jul 02 21:19:13.801592 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:13.801554    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dnrb6"
Jul 02 21:19:13.802078 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:13.802059    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dnrb6"
Jul 02 21:19:14.009040 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:14.009012    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:19:14.009197 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:14.009049    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:19:14.009197 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:14.009133    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxcm5" podUID="28a7f381-3db6-43e5-9690-d409f048dcf2"
Jul 02 21:19:14.009314 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:14.009267    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gpccf" podUID="3a6e6de1-8b1b-478e-8dce-996e81502c1e"
Jul 02 21:19:14.153969 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:14.153874    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2725k" event={"ID":"bf253477-5a39-4648-9b18-4282286ab281","Type":"ContainerStarted","Data":"89db7ebe2e665c4052850957e76601ef60770f3e2f9e23325a8c5f52b1ac9d86"}
Jul 02 21:19:15.009099 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:15.009065    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:19:15.009268 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:15.009207    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9tf7" podUID="4076e581-5038-4786-9565-c72be1618034"
Jul 02 21:19:16.009543 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:16.009505    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:19:16.010193 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:16.009504    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:19:16.010193 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:16.009665    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxcm5" podUID="28a7f381-3db6-43e5-9690-d409f048dcf2"
Jul 02 21:19:16.010193 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:16.009704    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gpccf" podUID="3a6e6de1-8b1b-478e-8dce-996e81502c1e"
Jul 02 21:19:17.009341 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:17.009144    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:19:17.009439 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:17.009416    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9tf7" podUID="4076e581-5038-4786-9565-c72be1618034"
Jul 02 21:19:17.164365 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:17.164335    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2725k" event={"ID":"bf253477-5a39-4648-9b18-4282286ab281","Type":"ContainerStarted","Data":"57405252a30be05adba70ea839a1e646f11a021d3223e6f15194eddd8f9ecd7c"}
Jul 02 21:19:17.164944 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:17.164650    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:19:17.166462 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:17.166442    2575 generic.go:356] "Generic (PLEG): container finished" podID="4b7e06c9-44c9-486b-8483-d71693ccc112" containerID="219021bf4392a394f98c4e8398ba4e845dae078fa03d1e0384e813deeedce3d0" exitCode=0
Jul 02 21:19:17.166555 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:17.166474    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rfbk7" event={"ID":"4b7e06c9-44c9-486b-8483-d71693ccc112","Type":"ContainerDied","Data":"219021bf4392a394f98c4e8398ba4e845dae078fa03d1e0384e813deeedce3d0"}
Jul 02 21:19:17.180613 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:17.180596    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:19:17.198485 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:17.198446    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2725k" podStartSLOduration=9.677567457 podStartE2EDuration="28.198434848s" totalImagesPullingTime="18.520867391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:17.197960354 +0000 UTC m=+28.824462230" watchObservedRunningTime="2026-07-02 21:19:17.198434848 +0000 UTC m=+28.824936724"
Jul 02 21:19:18.009304 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:18.009275    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:19:18.009304 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:18.009303    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:19:18.009595 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:18.009401    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxcm5" podUID="28a7f381-3db6-43e5-9690-d409f048dcf2"
Jul 02 21:19:18.009595 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:18.009518    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gpccf" podUID="3a6e6de1-8b1b-478e-8dce-996e81502c1e"
Jul 02 21:19:18.169536 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:18.169507    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:19:18.169961 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:18.169598    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:19:18.183152 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:18.183130    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:19:18.551632 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:18.551562    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pxcm5"]
Jul 02 21:19:18.551769 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:18.551695    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:19:18.551821 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:18.551792    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxcm5" podUID="28a7f381-3db6-43e5-9690-d409f048dcf2"
Jul 02 21:19:18.553945 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:18.553921    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b9tf7"]
Jul 02 21:19:18.554049 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:18.554012    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:19:18.554094 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:18.554081    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9tf7" podUID="4076e581-5038-4786-9565-c72be1618034"
Jul 02 21:19:18.569818 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:18.569792    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gpccf"]
Jul 02 21:19:18.569890 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:18.569878    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:19:18.569979 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:18.569963    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gpccf" podUID="3a6e6de1-8b1b-478e-8dce-996e81502c1e"
Jul 02 21:19:19.173388 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:19.173128    2575 generic.go:356] "Generic (PLEG): container finished" podID="4b7e06c9-44c9-486b-8483-d71693ccc112" containerID="038c2408ef19d79532cb2dbbe418718fa177f1c29323fcbdbf2f173fc28e8370" exitCode=0
Jul 02 21:19:19.173823 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:19.173200    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rfbk7" event={"ID":"4b7e06c9-44c9-486b-8483-d71693ccc112","Type":"ContainerDied","Data":"038c2408ef19d79532cb2dbbe418718fa177f1c29323fcbdbf2f173fc28e8370"}
Jul 02 21:19:19.339618 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:19.339593    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dnrb6"
Jul 02 21:19:19.339763 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:19.339694    2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 02 21:19:19.340123 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:19.340100    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dnrb6"
Jul 02 21:19:20.009269 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:20.009242    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:19:20.009455 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:20.009242    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:19:20.009455 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:20.009342    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9tf7" podUID="4076e581-5038-4786-9565-c72be1618034"
Jul 02 21:19:20.009536 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:20.009456    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gpccf" podUID="3a6e6de1-8b1b-478e-8dce-996e81502c1e"
Jul 02 21:19:20.009536 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:20.009242    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:19:20.009608 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:20.009565    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxcm5" podUID="28a7f381-3db6-43e5-9690-d409f048dcf2"
Jul 02 21:19:21.182712 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:21.182677    2575 generic.go:356] "Generic (PLEG): container finished" podID="4b7e06c9-44c9-486b-8483-d71693ccc112" containerID="4ee0522806b725033d4fb422acfa382b5ed3b16d4a19a7825f65fde459f40de4" exitCode=0
Jul 02 21:19:21.183352 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:21.182753    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rfbk7" event={"ID":"4b7e06c9-44c9-486b-8483-d71693ccc112","Type":"ContainerDied","Data":"4ee0522806b725033d4fb422acfa382b5ed3b16d4a19a7825f65fde459f40de4"}
Jul 02 21:19:22.009110 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:22.009080    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:19:22.009297 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:22.009127    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:19:22.009297 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:22.009217    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:19:22.009297 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:22.009235    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxcm5" podUID="28a7f381-3db6-43e5-9690-d409f048dcf2"
Jul 02 21:19:22.009449 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:22.009296    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9tf7" podUID="4076e581-5038-4786-9565-c72be1618034"
Jul 02 21:19:22.009449 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:22.009370    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gpccf" podUID="3a6e6de1-8b1b-478e-8dce-996e81502c1e"
Jul 02 21:19:22.622493 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:22.622457    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret\") pod \"global-pull-secret-syncer-gpccf\" (UID: \"3a6e6de1-8b1b-478e-8dce-996e81502c1e\") " pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:19:22.623037 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:22.622508    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs\") pod \"network-metrics-daemon-pxcm5\" (UID: \"28a7f381-3db6-43e5-9690-d409f048dcf2\") " pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:19:22.623037 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:22.622636    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:22.623037 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:22.622718    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret podName:3a6e6de1-8b1b-478e-8dce-996e81502c1e nodeName:}" failed. No retries permitted until 2026-07-02 21:19:54.622699075 +0000 UTC m=+66.249200938 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret") pod "global-pull-secret-syncer-gpccf" (UID: "3a6e6de1-8b1b-478e-8dce-996e81502c1e") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:22.623037 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:22.622643    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:22.623037 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:22.622780    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs podName:28a7f381-3db6-43e5-9690-d409f048dcf2 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:54.622764058 +0000 UTC m=+66.249265917 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs") pod "network-metrics-daemon-pxcm5" (UID: "28a7f381-3db6-43e5-9690-d409f048dcf2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:22.723081 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:22.723047    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjksx\" (UniqueName: \"kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx\") pod \"network-check-target-b9tf7\" (UID: \"4076e581-5038-4786-9565-c72be1618034\") " pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:19:22.723270 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:22.723220    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:19:22.723270 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:22.723242    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:19:22.723270 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:22.723254    2575 projected.go:196] Error preparing data for projected volume kube-api-access-qjksx for pod openshift-network-diagnostics/network-check-target-b9tf7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:22.723396 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:22.723314    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx podName:4076e581-5038-4786-9565-c72be1618034 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:54.723296334 +0000 UTC m=+66.349798196 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qjksx" (UniqueName: "kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx") pod "network-check-target-b9tf7" (UID: "4076e581-5038-4786-9565-c72be1618034") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:24.009392 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.009206    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:19:24.009766 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.009238    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:19:24.009766 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.009247    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:19:24.009766 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:24.009574    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gpccf" podUID="3a6e6de1-8b1b-478e-8dce-996e81502c1e"
Jul 02 21:19:24.009766 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:24.009460    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b9tf7" podUID="4076e581-5038-4786-9565-c72be1618034"
Jul 02 21:19:24.009766 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:24.009641    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxcm5" podUID="28a7f381-3db6-43e5-9690-d409f048dcf2"
Jul 02 21:19:24.080602 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.080582    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-227.ec2.internal" event="NodeReady"
Jul 02 21:19:24.080726 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.080715    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 02 21:19:24.130399 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.130376    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-647f4fc976-pzbv8"]
Jul 02 21:19:24.132644 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.132621    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.143773 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.143753    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 02 21:19:24.143901 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.143752    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 02 21:19:24.143901 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.143850    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7gtpt\""
Jul 02 21:19:24.144009 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.143758    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 02 21:19:24.153375 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.153352    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 02 21:19:24.162031 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.161956    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4xxrj"]
Jul 02 21:19:24.163649 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.163630    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4xxrj"
Jul 02 21:19:24.165684 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.165662    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7vjmt"]
Jul 02 21:19:24.166513 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.166404    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jhvvb\""
Jul 02 21:19:24.166615 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.166598    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 02 21:19:24.166688 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.166625    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 02 21:19:24.166742 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.166626    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 02 21:19:24.167406 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.167385    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7vjmt"
Jul 02 21:19:24.172681 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.172663    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-647f4fc976-pzbv8"]
Jul 02 21:19:24.172808 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.172791    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 02 21:19:24.172892 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.172723    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 02 21:19:24.172988 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.172915    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qs2b2\""
Jul 02 21:19:24.194987 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.194965    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7vjmt"]
Jul 02 21:19:24.195668 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.195651    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4xxrj"]
Jul 02 21:19:24.237236 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.234743    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-bound-sa-token\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.237236 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.234843    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdnjn\" (UniqueName: \"kubernetes.io/projected/0c7a0d2a-4e86-46aa-9190-a0ce20694c85-kube-api-access-zdnjn\") pod \"ingress-canary-4xxrj\" (UID: \"0c7a0d2a-4e86-46aa-9190-a0ce20694c85\") " pod="openshift-ingress-canary/ingress-canary-4xxrj"
Jul 02 21:19:24.237236 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.234886    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.237236 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.234921    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vrss\" (UniqueName: \"kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-kube-api-access-6vrss\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.237236 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.234953    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d8e442e-419a-417c-bb3a-7227bf8e964f-ca-trust-extracted\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.237236 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.235012    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d8e442e-419a-417c-bb3a-7227bf8e964f-trusted-ca\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.237236 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.235046    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d8e442e-419a-417c-bb3a-7227bf8e964f-image-registry-private-configuration\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.237236 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.235071    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d8e442e-419a-417c-bb3a-7227bf8e964f-installation-pull-secrets\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.237236 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.235135    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c7a0d2a-4e86-46aa-9190-a0ce20694c85-cert\") pod \"ingress-canary-4xxrj\" (UID: \"0c7a0d2a-4e86-46aa-9190-a0ce20694c85\") " pod="openshift-ingress-canary/ingress-canary-4xxrj"
Jul 02 21:19:24.237236 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.235208    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-certificates\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.335719 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.335686    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5609d59-a06f-4db8-b564-a0ff92bd58b3-metrics-tls\") pod \"dns-default-7vjmt\" (UID: \"c5609d59-a06f-4db8-b564-a0ff92bd58b3\") " pod="openshift-dns/dns-default-7vjmt"
Jul 02 21:19:24.335879 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.335746    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-bound-sa-token\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.335879 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.335779    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdnjn\" (UniqueName: \"kubernetes.io/projected/0c7a0d2a-4e86-46aa-9190-a0ce20694c85-kube-api-access-zdnjn\") pod \"ingress-canary-4xxrj\" (UID: \"0c7a0d2a-4e86-46aa-9190-a0ce20694c85\") " pod="openshift-ingress-canary/ingress-canary-4xxrj"
Jul 02 21:19:24.335879 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.335801    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.335879 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.335828    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vrss\" (UniqueName: \"kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-kube-api-access-6vrss\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.335879 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.335852    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5609d59-a06f-4db8-b564-a0ff92bd58b3-tmp-dir\") pod \"dns-default-7vjmt\" (UID: \"c5609d59-a06f-4db8-b564-a0ff92bd58b3\") " pod="openshift-dns/dns-default-7vjmt"
Jul 02 21:19:24.335879 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.335872    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d8e442e-419a-417c-bb3a-7227bf8e964f-ca-trust-extracted\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.336195 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.335913    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d8e442e-419a-417c-bb3a-7227bf8e964f-trusted-ca\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.336195 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.335941    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d8e442e-419a-417c-bb3a-7227bf8e964f-image-registry-private-configuration\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.336195 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:24.335948    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 21:19:24.336195 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:24.335968    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-647f4fc976-pzbv8: secret "image-registry-tls" not found
Jul 02 21:19:24.336195 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:24.336029    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls podName:8d8e442e-419a-417c-bb3a-7227bf8e964f nodeName:}" failed. No retries permitted until 2026-07-02 21:19:24.836008079 +0000 UTC m=+36.462509934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls") pod "image-registry-647f4fc976-pzbv8" (UID: "8d8e442e-419a-417c-bb3a-7227bf8e964f") : secret "image-registry-tls" not found
Jul 02 21:19:24.336195 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.335966    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d8e442e-419a-417c-bb3a-7227bf8e964f-installation-pull-secrets\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.336483 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.336222    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c7a0d2a-4e86-46aa-9190-a0ce20694c85-cert\") pod \"ingress-canary-4xxrj\" (UID: \"0c7a0d2a-4e86-46aa-9190-a0ce20694c85\") " pod="openshift-ingress-canary/ingress-canary-4xxrj"
Jul 02 21:19:24.336483 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.336278    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6v2b\" (UniqueName: \"kubernetes.io/projected/c5609d59-a06f-4db8-b564-a0ff92bd58b3-kube-api-access-g6v2b\") pod \"dns-default-7vjmt\" (UID: \"c5609d59-a06f-4db8-b564-a0ff92bd58b3\") " pod="openshift-dns/dns-default-7vjmt"
Jul 02 21:19:24.336483 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.336321    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-certificates\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.336483 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.336343    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c5609d59-a06f-4db8-b564-a0ff92bd58b3-config-volume\") pod \"dns-default-7vjmt\" (UID: \"c5609d59-a06f-4db8-b564-a0ff92bd58b3\") " pod="openshift-dns/dns-default-7vjmt"
Jul 02 21:19:24.336483 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.336394    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d8e442e-419a-417c-bb3a-7227bf8e964f-ca-trust-extracted\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.336854 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.336834    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-certificates\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.337050 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.337019    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d8e442e-419a-417c-bb3a-7227bf8e964f-trusted-ca\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.340319 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.340297    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d8e442e-419a-417c-bb3a-7227bf8e964f-image-registry-private-configuration\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.340422 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.340301    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d8e442e-419a-417c-bb3a-7227bf8e964f-installation-pull-secrets\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.340422 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.340352    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c7a0d2a-4e86-46aa-9190-a0ce20694c85-cert\") pod \"ingress-canary-4xxrj\" (UID: \"0c7a0d2a-4e86-46aa-9190-a0ce20694c85\") " pod="openshift-ingress-canary/ingress-canary-4xxrj"
Jul 02 21:19:24.349059 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.349012    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdnjn\" (UniqueName: \"kubernetes.io/projected/0c7a0d2a-4e86-46aa-9190-a0ce20694c85-kube-api-access-zdnjn\") pod \"ingress-canary-4xxrj\" (UID: \"0c7a0d2a-4e86-46aa-9190-a0ce20694c85\") " pod="openshift-ingress-canary/ingress-canary-4xxrj"
Jul 02 21:19:24.350833 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.350813    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-bound-sa-token\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.351747 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.351729    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vrss\" (UniqueName: \"kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-kube-api-access-6vrss\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.437011 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.436917    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5609d59-a06f-4db8-b564-a0ff92bd58b3-tmp-dir\") pod \"dns-default-7vjmt\" (UID: \"c5609d59-a06f-4db8-b564-a0ff92bd58b3\") " pod="openshift-dns/dns-default-7vjmt"
Jul 02 21:19:24.437156 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.437110    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6v2b\" (UniqueName: \"kubernetes.io/projected/c5609d59-a06f-4db8-b564-a0ff92bd58b3-kube-api-access-g6v2b\") pod \"dns-default-7vjmt\" (UID: \"c5609d59-a06f-4db8-b564-a0ff92bd58b3\") " pod="openshift-dns/dns-default-7vjmt"
Jul 02 21:19:24.437156 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.437150    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c5609d59-a06f-4db8-b564-a0ff92bd58b3-config-volume\") pod \"dns-default-7vjmt\" (UID: \"c5609d59-a06f-4db8-b564-a0ff92bd58b3\") " pod="openshift-dns/dns-default-7vjmt"
Jul 02 21:19:24.437284 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.437208    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5609d59-a06f-4db8-b564-a0ff92bd58b3-metrics-tls\") pod \"dns-default-7vjmt\" (UID: \"c5609d59-a06f-4db8-b564-a0ff92bd58b3\") " pod="openshift-dns/dns-default-7vjmt"
Jul 02 21:19:24.437284 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.437260    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5609d59-a06f-4db8-b564-a0ff92bd58b3-tmp-dir\") pod \"dns-default-7vjmt\" (UID: \"c5609d59-a06f-4db8-b564-a0ff92bd58b3\") " pod="openshift-dns/dns-default-7vjmt"
Jul 02 21:19:24.437373 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:24.437308    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 21:19:24.437373 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:24.437366    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5609d59-a06f-4db8-b564-a0ff92bd58b3-metrics-tls podName:c5609d59-a06f-4db8-b564-a0ff92bd58b3 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:24.937349559 +0000 UTC m=+36.563851428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5609d59-a06f-4db8-b564-a0ff92bd58b3-metrics-tls") pod "dns-default-7vjmt" (UID: "c5609d59-a06f-4db8-b564-a0ff92bd58b3") : secret "dns-default-metrics-tls" not found
Jul 02 21:19:24.437734 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.437715    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c5609d59-a06f-4db8-b564-a0ff92bd58b3-config-volume\") pod \"dns-default-7vjmt\" (UID: \"c5609d59-a06f-4db8-b564-a0ff92bd58b3\") " pod="openshift-dns/dns-default-7vjmt"
Jul 02 21:19:24.448998 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.448978    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6v2b\" (UniqueName: \"kubernetes.io/projected/c5609d59-a06f-4db8-b564-a0ff92bd58b3-kube-api-access-g6v2b\") pod \"dns-default-7vjmt\" (UID: \"c5609d59-a06f-4db8-b564-a0ff92bd58b3\") " pod="openshift-dns/dns-default-7vjmt"
Jul 02 21:19:24.474986 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.474960    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4xxrj"
Jul 02 21:19:24.840112 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.840033    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:24.840278 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:24.840213    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 21:19:24.840278 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:24.840230    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-647f4fc976-pzbv8: secret "image-registry-tls" not found
Jul 02 21:19:24.840374 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:24.840284    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls podName:8d8e442e-419a-417c-bb3a-7227bf8e964f nodeName:}" failed. No retries permitted until 2026-07-02 21:19:25.840269643 +0000 UTC m=+37.466771497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls") pod "image-registry-647f4fc976-pzbv8" (UID: "8d8e442e-419a-417c-bb3a-7227bf8e964f") : secret "image-registry-tls" not found
Jul 02 21:19:24.941427 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:24.941394    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5609d59-a06f-4db8-b564-a0ff92bd58b3-metrics-tls\") pod \"dns-default-7vjmt\" (UID: \"c5609d59-a06f-4db8-b564-a0ff92bd58b3\") " pod="openshift-dns/dns-default-7vjmt"
Jul 02 21:19:24.941612 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:24.941558    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 21:19:24.941677 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:24.941629    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5609d59-a06f-4db8-b564-a0ff92bd58b3-metrics-tls podName:c5609d59-a06f-4db8-b564-a0ff92bd58b3 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:25.941610648 +0000 UTC m=+37.568112502 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5609d59-a06f-4db8-b564-a0ff92bd58b3-metrics-tls") pod "dns-default-7vjmt" (UID: "c5609d59-a06f-4db8-b564-a0ff92bd58b3") : secret "dns-default-metrics-tls" not found
Jul 02 21:19:25.181938 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:25.181916    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4xxrj"]
Jul 02 21:19:25.184597 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:19:25.184570    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c7a0d2a_4e86_46aa_9190_a0ce20694c85.slice/crio-817562e232b187e3522dd9bc421d337f9417fb4b638e1986e7405db7b0ba1265 WatchSource:0}: Error finding container 817562e232b187e3522dd9bc421d337f9417fb4b638e1986e7405db7b0ba1265: Status 404 returned error can't find the container with id 817562e232b187e3522dd9bc421d337f9417fb4b638e1986e7405db7b0ba1265
Jul 02 21:19:25.193865 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:25.193842    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4xxrj" event={"ID":"0c7a0d2a-4e86-46aa-9190-a0ce20694c85","Type":"ContainerStarted","Data":"817562e232b187e3522dd9bc421d337f9417fb4b638e1986e7405db7b0ba1265"}
Jul 02 21:19:25.848031 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:25.847954    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:25.848191 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:25.848121    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 21:19:25.848191 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:25.848143    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-647f4fc976-pzbv8: secret "image-registry-tls" not found
Jul 02 21:19:25.848308 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:25.848226    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls podName:8d8e442e-419a-417c-bb3a-7227bf8e964f nodeName:}" failed. No retries permitted until 2026-07-02 21:19:27.848207028 +0000 UTC m=+39.474708885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls") pod "image-registry-647f4fc976-pzbv8" (UID: "8d8e442e-419a-417c-bb3a-7227bf8e964f") : secret "image-registry-tls" not found
Jul 02 21:19:25.948774 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:25.948744    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5609d59-a06f-4db8-b564-a0ff92bd58b3-metrics-tls\") pod \"dns-default-7vjmt\" (UID: \"c5609d59-a06f-4db8-b564-a0ff92bd58b3\") " pod="openshift-dns/dns-default-7vjmt"
Jul 02 21:19:25.948936 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:25.948891    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 21:19:25.948999 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:25.948965    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5609d59-a06f-4db8-b564-a0ff92bd58b3-metrics-tls podName:c5609d59-a06f-4db8-b564-a0ff92bd58b3 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:27.948945283 +0000 UTC m=+39.575447136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5609d59-a06f-4db8-b564-a0ff92bd58b3-metrics-tls") pod "dns-default-7vjmt" (UID: "c5609d59-a06f-4db8-b564-a0ff92bd58b3") : secret "dns-default-metrics-tls" not found
Jul 02 21:19:26.009390 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:26.009363    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:19:26.009526 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:26.009495    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:19:26.009600 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:26.009528    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:19:26.013750 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:26.013728    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 21:19:26.013857 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:26.013754    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 21:19:26.013857 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:26.013768    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 21:19:26.013857 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:26.013839    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zt75x\""
Jul 02 21:19:26.013857 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:26.013768    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 21:19:26.014103 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:26.014084    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zvz49\""
Jul 02 21:19:26.199769 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:26.199737    2575 generic.go:356] "Generic (PLEG): container finished" podID="4b7e06c9-44c9-486b-8483-d71693ccc112" containerID="eff20ad22c06edd6c78a8588a892028c241dfd030ce82199f2fd1d9bcfb83783" exitCode=0
Jul 02 21:19:26.200163 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:26.199786    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rfbk7" event={"ID":"4b7e06c9-44c9-486b-8483-d71693ccc112","Type":"ContainerDied","Data":"eff20ad22c06edd6c78a8588a892028c241dfd030ce82199f2fd1d9bcfb83783"}
Jul 02 21:19:27.207364 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:27.207102    2575 generic.go:356] "Generic (PLEG): container finished" podID="4b7e06c9-44c9-486b-8483-d71693ccc112" containerID="a150e5fb2e4ae651568bc07d3266595a416919e14985aabc39403e40fec00ff0" exitCode=0
Jul 02 21:19:27.207364 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:27.207200    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rfbk7" event={"ID":"4b7e06c9-44c9-486b-8483-d71693ccc112","Type":"ContainerDied","Data":"a150e5fb2e4ae651568bc07d3266595a416919e14985aabc39403e40fec00ff0"}
Jul 02 21:19:27.865830 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:27.865748    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:27.866014 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:27.865899    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 21:19:27.866014 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:27.865918    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-647f4fc976-pzbv8: secret "image-registry-tls" not found
Jul 02 21:19:27.866014 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:27.865972    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls podName:8d8e442e-419a-417c-bb3a-7227bf8e964f nodeName:}" failed. No retries permitted until 2026-07-02 21:19:31.865957983 +0000 UTC m=+43.492459841 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls") pod "image-registry-647f4fc976-pzbv8" (UID: "8d8e442e-419a-417c-bb3a-7227bf8e964f") : secret "image-registry-tls" not found
Jul 02 21:19:27.966589 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:27.966563    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5609d59-a06f-4db8-b564-a0ff92bd58b3-metrics-tls\") pod \"dns-default-7vjmt\" (UID: \"c5609d59-a06f-4db8-b564-a0ff92bd58b3\") " pod="openshift-dns/dns-default-7vjmt"
Jul 02 21:19:27.966702 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:27.966689    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 21:19:27.966748 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:27.966740    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5609d59-a06f-4db8-b564-a0ff92bd58b3-metrics-tls podName:c5609d59-a06f-4db8-b564-a0ff92bd58b3 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:31.966727995 +0000 UTC m=+43.593229849 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5609d59-a06f-4db8-b564-a0ff92bd58b3-metrics-tls") pod "dns-default-7vjmt" (UID: "c5609d59-a06f-4db8-b564-a0ff92bd58b3") : secret "dns-default-metrics-tls" not found
Jul 02 21:19:28.210502 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:28.210472    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4xxrj" event={"ID":"0c7a0d2a-4e86-46aa-9190-a0ce20694c85","Type":"ContainerStarted","Data":"83c0a585532f10114f4ff2ba48c3e723c7337c8a81044896262edf8392352a25"}
Jul 02 21:19:28.213497 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:28.213474    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rfbk7" event={"ID":"4b7e06c9-44c9-486b-8483-d71693ccc112","Type":"ContainerStarted","Data":"2688e8e526b20f70c36d7b1761f9a723e933a7b85dae278f1b04c0666271cb94"}
Jul 02 21:19:28.233601 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:28.233562    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4xxrj" podStartSLOduration=1.931381751 podStartE2EDuration="4.233548768s" totalImagesPullingTime="2.302167017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:28.233516305 +0000 UTC m=+39.860018180" watchObservedRunningTime="2026-07-02 21:19:28.233548768 +0000 UTC m=+39.860050643"
Jul 02 21:19:28.466350 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:28.466270    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:28.595478 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:28.595448    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:28.973872 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:28.973846    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:29.050775 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:29.050744    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:30.637794 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.637736    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rfbk7" podStartSLOduration=11.878271539 podStartE2EDuration="41.637716281s" totalImagesPullingTime="29.759444742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:28.279062628 +0000 UTC m=+39.905564503" watchObservedRunningTime="2026-07-02 21:19:30.637716281 +0000 UTC m=+42.264218160"
Jul 02 21:19:30.638761 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.638736    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-7tvx2"]
Jul 02 21:19:30.672490 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.672463    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-pbcv9"]
Jul 02 21:19:30.672616 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.672598    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-7tvx2"
Jul 02 21:19:30.675978 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.675960    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-tcm9m\""
Jul 02 21:19:30.676920 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.676898    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 02 21:19:30.677510 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.677491    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 02 21:19:30.697330 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.697309    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-pbcv9"]
Jul 02 21:19:30.697401 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.697335    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-7tvx2"]
Jul 02 21:19:30.697443 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.697399    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-pbcv9"
Jul 02 21:19:30.699975 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.699957    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 02 21:19:30.700058 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.700000    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 02 21:19:30.700113 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.700003    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-wwvmh\""
Jul 02 21:19:30.700196 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.700182    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 02 21:19:30.700446 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.700432    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 02 21:19:30.786028 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.785998    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/78977169-1717-4b34-b8ac-7f4562f6be64-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-pbcv9\" (UID: \"78977169-1717-4b34-b8ac-7f4562f6be64\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-pbcv9"
Jul 02 21:19:30.786131 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.786043    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/78977169-1717-4b34-b8ac-7f4562f6be64-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-pbcv9\" (UID: \"78977169-1717-4b34-b8ac-7f4562f6be64\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-pbcv9"
Jul 02 21:19:30.786131 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.786069    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dktl9\" (UniqueName: \"kubernetes.io/projected/78977169-1717-4b34-b8ac-7f4562f6be64-kube-api-access-dktl9\") pod \"kube-storage-version-migrator-operator-846bf668f7-pbcv9\" (UID: \"78977169-1717-4b34-b8ac-7f4562f6be64\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-pbcv9"
Jul 02 21:19:30.786265 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.786196    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v96ss\" (UniqueName: \"kubernetes.io/projected/91aa7825-8fff-45f4-9a55-bef98c760952-kube-api-access-v96ss\") pod \"volume-data-source-validator-5d456fdbcd-7tvx2\" (UID: \"91aa7825-8fff-45f4-9a55-bef98c760952\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-7tvx2"
Jul 02 21:19:30.858832 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.858810    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:30.886813 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.886788    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/78977169-1717-4b34-b8ac-7f4562f6be64-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-pbcv9\" (UID: \"78977169-1717-4b34-b8ac-7f4562f6be64\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-pbcv9"
Jul 02 21:19:30.886910 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.886830    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/78977169-1717-4b34-b8ac-7f4562f6be64-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-pbcv9\" (UID: \"78977169-1717-4b34-b8ac-7f4562f6be64\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-pbcv9"
Jul 02 21:19:30.886910 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.886849    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dktl9\" (UniqueName: \"kubernetes.io/projected/78977169-1717-4b34-b8ac-7f4562f6be64-kube-api-access-dktl9\") pod \"kube-storage-version-migrator-operator-846bf668f7-pbcv9\" (UID: \"78977169-1717-4b34-b8ac-7f4562f6be64\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-pbcv9"
Jul 02 21:19:30.886910 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.886907    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v96ss\" (UniqueName: \"kubernetes.io/projected/91aa7825-8fff-45f4-9a55-bef98c760952-kube-api-access-v96ss\") pod \"volume-data-source-validator-5d456fdbcd-7tvx2\" (UID: \"91aa7825-8fff-45f4-9a55-bef98c760952\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-7tvx2"
Jul 02 21:19:30.887376 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.887356    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/78977169-1717-4b34-b8ac-7f4562f6be64-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-pbcv9\" (UID: \"78977169-1717-4b34-b8ac-7f4562f6be64\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-pbcv9"
Jul 02 21:19:30.890063 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.890020    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/78977169-1717-4b34-b8ac-7f4562f6be64-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-pbcv9\" (UID: \"78977169-1717-4b34-b8ac-7f4562f6be64\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-pbcv9"
Jul 02 21:19:30.897178 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.897151    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dktl9\" (UniqueName: \"kubernetes.io/projected/78977169-1717-4b34-b8ac-7f4562f6be64-kube-api-access-dktl9\") pod \"kube-storage-version-migrator-operator-846bf668f7-pbcv9\" (UID: \"78977169-1717-4b34-b8ac-7f4562f6be64\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-pbcv9"
Jul 02 21:19:30.897260 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.897184    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v96ss\" (UniqueName: \"kubernetes.io/projected/91aa7825-8fff-45f4-9a55-bef98c760952-kube-api-access-v96ss\") pod \"volume-data-source-validator-5d456fdbcd-7tvx2\" (UID: \"91aa7825-8fff-45f4-9a55-bef98c760952\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-7tvx2"
Jul 02 21:19:30.959820 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.959800    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:30.982728 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:30.982713    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-7tvx2"
Jul 02 21:19:31.005011 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:31.004993    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-pbcv9"
Jul 02 21:19:31.127472 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:31.127447    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-7tvx2"]
Jul 02 21:19:31.130438 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:19:31.130402    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91aa7825_8fff_45f4_9a55_bef98c760952.slice/crio-1716c49321c31e762f5f13da74db4f7b2eb2c692170e1ee647f5e1be31f4e60c WatchSource:0}: Error finding container 1716c49321c31e762f5f13da74db4f7b2eb2c692170e1ee647f5e1be31f4e60c: Status 404 returned error can't find the container with id 1716c49321c31e762f5f13da74db4f7b2eb2c692170e1ee647f5e1be31f4e60c
Jul 02 21:19:31.148554 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:31.148530    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-pbcv9"]
Jul 02 21:19:31.151293 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:19:31.151272    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78977169_1717_4b34_b8ac_7f4562f6be64.slice/crio-9e16a426888c3d2a6ff2ad5e8c0b99cc1daa6eb3f94735f49958a1b1a42c9467 WatchSource:0}: Error finding container 9e16a426888c3d2a6ff2ad5e8c0b99cc1daa6eb3f94735f49958a1b1a42c9467: Status 404 returned error can't find the container with id 9e16a426888c3d2a6ff2ad5e8c0b99cc1daa6eb3f94735f49958a1b1a42c9467
Jul 02 21:19:31.220402 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:31.220376    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-pbcv9" event={"ID":"78977169-1717-4b34-b8ac-7f4562f6be64","Type":"ContainerStarted","Data":"9e16a426888c3d2a6ff2ad5e8c0b99cc1daa6eb3f94735f49958a1b1a42c9467"}
Jul 02 21:19:31.221430 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:31.221411    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-7tvx2" event={"ID":"91aa7825-8fff-45f4-9a55-bef98c760952","Type":"ContainerStarted","Data":"1716c49321c31e762f5f13da74db4f7b2eb2c692170e1ee647f5e1be31f4e60c"}
Jul 02 21:19:31.893743 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:31.893705    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:31.894203 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:31.893890    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 21:19:31.894203 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:31.893911    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-647f4fc976-pzbv8: secret "image-registry-tls" not found
Jul 02 21:19:31.894203 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:31.893982    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls podName:8d8e442e-419a-417c-bb3a-7227bf8e964f nodeName:}" failed. No retries permitted until 2026-07-02 21:19:39.893960644 +0000 UTC m=+51.520462709 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls") pod "image-registry-647f4fc976-pzbv8" (UID: "8d8e442e-419a-417c-bb3a-7227bf8e964f") : secret "image-registry-tls" not found
Jul 02 21:19:31.995000 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:31.994968    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5609d59-a06f-4db8-b564-a0ff92bd58b3-metrics-tls\") pod \"dns-default-7vjmt\" (UID: \"c5609d59-a06f-4db8-b564-a0ff92bd58b3\") " pod="openshift-dns/dns-default-7vjmt"
Jul 02 21:19:31.995163 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:31.995140    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 21:19:31.995246 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:31.995233    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5609d59-a06f-4db8-b564-a0ff92bd58b3-metrics-tls podName:c5609d59-a06f-4db8-b564-a0ff92bd58b3 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:39.995212006 +0000 UTC m=+51.621713860 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5609d59-a06f-4db8-b564-a0ff92bd58b3-metrics-tls") pod "dns-default-7vjmt" (UID: "c5609d59-a06f-4db8-b564-a0ff92bd58b3") : secret "dns-default-metrics-tls" not found
Jul 02 21:19:33.063750 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:33.063717    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:33.162607 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:33.162573    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:33.228590 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:33.228559    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-7tvx2" event={"ID":"91aa7825-8fff-45f4-9a55-bef98c760952","Type":"ContainerStarted","Data":"94e02c131988e8f7e1f8ae120241d026e32c06cf45be57989fe47f44688f516f"}
Jul 02 21:19:33.255850 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:33.254448    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-7tvx2" podStartSLOduration=1.77744488 podStartE2EDuration="3.254422127s" totalImagesPullingTime="1.476977247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:33.253087818 +0000 UTC m=+44.879589695" watchObservedRunningTime="2026-07-02 21:19:33.254422127 +0000 UTC m=+44.880924012"
Jul 02 21:19:33.460315 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:33.460280    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:33.579701 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:33.579682    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:34.231850 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:34.231813    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-pbcv9" event={"ID":"78977169-1717-4b34-b8ac-7f4562f6be64","Type":"ContainerStarted","Data":"b2de0d017859b1c255d6ffaf122c2dfc57d9c044733f1e12efc4ff60ef0eedc1"}
Jul 02 21:19:34.251322 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:34.251282    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-pbcv9" podStartSLOduration=1.882752425 podStartE2EDuration="4.251268121s" totalImagesPullingTime="2.368515696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:34.250610952 +0000 UTC m=+45.877112827" watchObservedRunningTime="2026-07-02 21:19:34.251268121 +0000 UTC m=+45.877769996"
Jul 02 21:19:34.589146 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:34.589072    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:34.683005 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:34.682974    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:34.779797 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:34.779757    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:34.847398 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:34.847317    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:35.024374 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:35.024339    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:37.315025 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:37.314992    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:37.378096 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:37.378065    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:37.439646 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:37.439624    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:37.945894 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:37.945864    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pvf5d"]
Jul 02 21:19:37.991997 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:37.991965    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pvf5d"]
Jul 02 21:19:37.992136 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:37.992095    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:37.994757 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:37.994737    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 02 21:19:37.994757 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:37.994748    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 02 21:19:37.995867 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:37.995849    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m4mjq\""
Jul 02 21:19:37.995943 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:37.995866    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 02 21:19:37.996001 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:37.995943    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 02 21:19:38.003042 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:38.003018    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:38.065712 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:38.065687    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:38.144807 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:38.144786    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/06a47112-ec12-4baa-bd99-91b2d599254b-data-volume\") pod \"insights-runtime-extractor-pvf5d\" (UID: \"06a47112-ec12-4baa-bd99-91b2d599254b\") " pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:38.144899 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:38.144811    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06a47112-ec12-4baa-bd99-91b2d599254b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvf5d\" (UID: \"06a47112-ec12-4baa-bd99-91b2d599254b\") " pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:38.144899 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:38.144847    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/06a47112-ec12-4baa-bd99-91b2d599254b-crio-socket\") pod \"insights-runtime-extractor-pvf5d\" (UID: \"06a47112-ec12-4baa-bd99-91b2d599254b\") " pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:38.144993 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:38.144917    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06a47112-ec12-4baa-bd99-91b2d599254b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pvf5d\" (UID: \"06a47112-ec12-4baa-bd99-91b2d599254b\") " pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:38.144993 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:38.144934    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcrdq\" (UniqueName: \"kubernetes.io/projected/06a47112-ec12-4baa-bd99-91b2d599254b-kube-api-access-rcrdq\") pod \"insights-runtime-extractor-pvf5d\" (UID: \"06a47112-ec12-4baa-bd99-91b2d599254b\") " pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:38.245461 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:38.245378    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/06a47112-ec12-4baa-bd99-91b2d599254b-crio-socket\") pod \"insights-runtime-extractor-pvf5d\" (UID: \"06a47112-ec12-4baa-bd99-91b2d599254b\") " pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:38.245608 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:38.245461    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06a47112-ec12-4baa-bd99-91b2d599254b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pvf5d\" (UID: \"06a47112-ec12-4baa-bd99-91b2d599254b\") " pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:38.245608 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:38.245488    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcrdq\" (UniqueName: \"kubernetes.io/projected/06a47112-ec12-4baa-bd99-91b2d599254b-kube-api-access-rcrdq\") pod \"insights-runtime-extractor-pvf5d\" (UID: \"06a47112-ec12-4baa-bd99-91b2d599254b\") " pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:38.245608 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:38.245505    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/06a47112-ec12-4baa-bd99-91b2d599254b-crio-socket\") pod \"insights-runtime-extractor-pvf5d\" (UID: \"06a47112-ec12-4baa-bd99-91b2d599254b\") " pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:38.245608 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:38.245554    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/06a47112-ec12-4baa-bd99-91b2d599254b-data-volume\") pod \"insights-runtime-extractor-pvf5d\" (UID: \"06a47112-ec12-4baa-bd99-91b2d599254b\") " pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:38.245608 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:38.245585    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 21:19:38.245795 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:38.245585    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06a47112-ec12-4baa-bd99-91b2d599254b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvf5d\" (UID: \"06a47112-ec12-4baa-bd99-91b2d599254b\") " pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:38.245795 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:38.245677    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/06a47112-ec12-4baa-bd99-91b2d599254b-kube-rbac-proxy-cm podName:06a47112-ec12-4baa-bd99-91b2d599254b nodeName:}" failed. No retries permitted until 2026-07-02 21:19:38.745653191 +0000 UTC m=+50.372155064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/06a47112-ec12-4baa-bd99-91b2d599254b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pvf5d" (UID: "06a47112-ec12-4baa-bd99-91b2d599254b") : configmap "kube-rbac-proxy" not found
Jul 02 21:19:38.245795 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:38.245709    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:38.245902 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:38.245803    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06a47112-ec12-4baa-bd99-91b2d599254b-insights-runtime-extractor-tls podName:06a47112-ec12-4baa-bd99-91b2d599254b nodeName:}" failed. No retries permitted until 2026-07-02 21:19:38.745766681 +0000 UTC m=+50.372268548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/06a47112-ec12-4baa-bd99-91b2d599254b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pvf5d" (UID: "06a47112-ec12-4baa-bd99-91b2d599254b") : secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:38.245947 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:38.245934    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/06a47112-ec12-4baa-bd99-91b2d599254b-data-volume\") pod \"insights-runtime-extractor-pvf5d\" (UID: \"06a47112-ec12-4baa-bd99-91b2d599254b\") " pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:38.256462 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:38.256431    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcrdq\" (UniqueName: \"kubernetes.io/projected/06a47112-ec12-4baa-bd99-91b2d599254b-kube-api-access-rcrdq\") pod \"insights-runtime-extractor-pvf5d\" (UID: \"06a47112-ec12-4baa-bd99-91b2d599254b\") " pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:38.487073 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:38.487037    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:38.558362 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:38.558295    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:38.750989 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:38.750952    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06a47112-ec12-4baa-bd99-91b2d599254b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pvf5d\" (UID: \"06a47112-ec12-4baa-bd99-91b2d599254b\") " pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:38.751203 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:38.751028    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06a47112-ec12-4baa-bd99-91b2d599254b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvf5d\" (UID: \"06a47112-ec12-4baa-bd99-91b2d599254b\") " pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:38.751203 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:38.751088    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 21:19:38.751203 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:38.751164    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/06a47112-ec12-4baa-bd99-91b2d599254b-kube-rbac-proxy-cm podName:06a47112-ec12-4baa-bd99-91b2d599254b nodeName:}" failed. No retries permitted until 2026-07-02 21:19:39.751149486 +0000 UTC m=+51.377651341 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/06a47112-ec12-4baa-bd99-91b2d599254b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pvf5d" (UID: "06a47112-ec12-4baa-bd99-91b2d599254b") : configmap "kube-rbac-proxy" not found
Jul 02 21:19:38.751346 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:38.751226    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:38.751346 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:38.751304    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06a47112-ec12-4baa-bd99-91b2d599254b-insights-runtime-extractor-tls podName:06a47112-ec12-4baa-bd99-91b2d599254b nodeName:}" failed. No retries permitted until 2026-07-02 21:19:39.751286859 +0000 UTC m=+51.377788716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/06a47112-ec12-4baa-bd99-91b2d599254b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pvf5d" (UID: "06a47112-ec12-4baa-bd99-91b2d599254b") : secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:38.778052 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:38.778021    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:38.869098 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:38.869019    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:39.285100 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:39.285066    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:39.356858 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:39.356821    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:39.757916 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:39.757889    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06a47112-ec12-4baa-bd99-91b2d599254b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pvf5d\" (UID: \"06a47112-ec12-4baa-bd99-91b2d599254b\") " pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:39.758265 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:39.757939    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06a47112-ec12-4baa-bd99-91b2d599254b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvf5d\" (UID: \"06a47112-ec12-4baa-bd99-91b2d599254b\") " pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:39.758265 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:39.758029    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:39.758265 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:39.758077    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06a47112-ec12-4baa-bd99-91b2d599254b-insights-runtime-extractor-tls podName:06a47112-ec12-4baa-bd99-91b2d599254b nodeName:}" failed. No retries permitted until 2026-07-02 21:19:41.758064965 +0000 UTC m=+53.384566819 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/06a47112-ec12-4baa-bd99-91b2d599254b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pvf5d" (UID: "06a47112-ec12-4baa-bd99-91b2d599254b") : secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:39.758406 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:39.758386    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06a47112-ec12-4baa-bd99-91b2d599254b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pvf5d\" (UID: \"06a47112-ec12-4baa-bd99-91b2d599254b\") " pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:39.840921 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:39.840895    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:39.959397 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:39.959370    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:39.959517 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:39.959482    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 21:19:39.959517 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:39.959496    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-647f4fc976-pzbv8: secret "image-registry-tls" not found
Jul 02 21:19:39.959631 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:39.959549    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls podName:8d8e442e-419a-417c-bb3a-7227bf8e964f nodeName:}" failed. No retries permitted until 2026-07-02 21:19:55.959532891 +0000 UTC m=+67.586034747 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls") pod "image-registry-647f4fc976-pzbv8" (UID: "8d8e442e-419a-417c-bb3a-7227bf8e964f") : secret "image-registry-tls" not found
Jul 02 21:19:40.059882 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.059821    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5609d59-a06f-4db8-b564-a0ff92bd58b3-metrics-tls\") pod \"dns-default-7vjmt\" (UID: \"c5609d59-a06f-4db8-b564-a0ff92bd58b3\") " pod="openshift-dns/dns-default-7vjmt"
Jul 02 21:19:40.059986 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:40.059951    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 21:19:40.060030 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:40.059998    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5609d59-a06f-4db8-b564-a0ff92bd58b3-metrics-tls podName:c5609d59-a06f-4db8-b564-a0ff92bd58b3 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:56.059986049 +0000 UTC m=+67.686487903 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5609d59-a06f-4db8-b564-a0ff92bd58b3-metrics-tls") pod "dns-default-7vjmt" (UID: "c5609d59-a06f-4db8-b564-a0ff92bd58b3") : secret "dns-default-metrics-tls" not found
Jul 02 21:19:40.358702 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.358624    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-8gmq5"]
Jul 02 21:19:40.371984 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.371960    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-8gmq5"
Jul 02 21:19:40.372596 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.372578    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-8gmq5"]
Jul 02 21:19:40.374741 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.374722    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 02 21:19:40.374839 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.374813    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-w6mn7\""
Jul 02 21:19:40.374902 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.374868    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 02 21:19:40.374957 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.374905    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 02 21:19:40.375750 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.375707    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 02 21:19:40.400903 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.400881    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:40.464428 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.464401    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:40.564037 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.564008    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ee079266-209e-4bed-9054-946392b449c3-signing-cabundle\") pod \"service-ca-758f599c5-8gmq5\" (UID: \"ee079266-209e-4bed-9054-946392b449c3\") " pod="openshift-service-ca/service-ca-758f599c5-8gmq5"
Jul 02 21:19:40.564164 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.564112    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7945t\" (UniqueName: \"kubernetes.io/projected/ee079266-209e-4bed-9054-946392b449c3-kube-api-access-7945t\") pod \"service-ca-758f599c5-8gmq5\" (UID: \"ee079266-209e-4bed-9054-946392b449c3\") " pod="openshift-service-ca/service-ca-758f599c5-8gmq5"
Jul 02 21:19:40.564164 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.564151    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ee079266-209e-4bed-9054-946392b449c3-signing-key\") pod \"service-ca-758f599c5-8gmq5\" (UID: \"ee079266-209e-4bed-9054-946392b449c3\") " pod="openshift-service-ca/service-ca-758f599c5-8gmq5"
Jul 02 21:19:40.665454 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.665422    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ee079266-209e-4bed-9054-946392b449c3-signing-cabundle\") pod \"service-ca-758f599c5-8gmq5\" (UID: \"ee079266-209e-4bed-9054-946392b449c3\") " pod="openshift-service-ca/service-ca-758f599c5-8gmq5"
Jul 02 21:19:40.665616 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.665488    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7945t\" (UniqueName: \"kubernetes.io/projected/ee079266-209e-4bed-9054-946392b449c3-kube-api-access-7945t\") pod \"service-ca-758f599c5-8gmq5\" (UID: \"ee079266-209e-4bed-9054-946392b449c3\") " pod="openshift-service-ca/service-ca-758f599c5-8gmq5"
Jul 02 21:19:40.665616 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.665519    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ee079266-209e-4bed-9054-946392b449c3-signing-key\") pod \"service-ca-758f599c5-8gmq5\" (UID: \"ee079266-209e-4bed-9054-946392b449c3\") " pod="openshift-service-ca/service-ca-758f599c5-8gmq5"
Jul 02 21:19:40.666109 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.666083    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ee079266-209e-4bed-9054-946392b449c3-signing-cabundle\") pod \"service-ca-758f599c5-8gmq5\" (UID: \"ee079266-209e-4bed-9054-946392b449c3\") " pod="openshift-service-ca/service-ca-758f599c5-8gmq5"
Jul 02 21:19:40.667857 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.667838    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ee079266-209e-4bed-9054-946392b449c3-signing-key\") pod \"service-ca-758f599c5-8gmq5\" (UID: \"ee079266-209e-4bed-9054-946392b449c3\") " pod="openshift-service-ca/service-ca-758f599c5-8gmq5"
Jul 02 21:19:40.674192 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.674158    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7945t\" (UniqueName: \"kubernetes.io/projected/ee079266-209e-4bed-9054-946392b449c3-kube-api-access-7945t\") pod \"service-ca-758f599c5-8gmq5\" (UID: \"ee079266-209e-4bed-9054-946392b449c3\") " pod="openshift-service-ca/service-ca-758f599c5-8gmq5"
Jul 02 21:19:40.680991 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.680975    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-8gmq5"
Jul 02 21:19:40.750238 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.750117    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k4r2h_a0c7410f-98e2-4331-b24b-6eeab7f99401/dns-node-resolver/0.log"
Jul 02 21:19:40.814121 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:40.814093    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-8gmq5"]
Jul 02 21:19:40.816560 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:19:40.816535    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee079266_209e_4bed_9054_946392b449c3.slice/crio-7d6a8888a79ebe47b691b40cce6f2b788bff76dc5928e70d4875b27b4f0fc121 WatchSource:0}: Error finding container 7d6a8888a79ebe47b691b40cce6f2b788bff76dc5928e70d4875b27b4f0fc121: Status 404 returned error can't find the container with id 7d6a8888a79ebe47b691b40cce6f2b788bff76dc5928e70d4875b27b4f0fc121
Jul 02 21:19:41.249298 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:41.249265    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-8gmq5" event={"ID":"ee079266-209e-4bed-9054-946392b449c3","Type":"ContainerStarted","Data":"7d6a8888a79ebe47b691b40cce6f2b788bff76dc5928e70d4875b27b4f0fc121"}
Jul 02 21:19:41.405152 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:41.405119    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:41.550124 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:41.550058    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9zmmn_83c0ff63-726d-4bb4-b4f6-0810ef1af2b2/node-ca/0.log"
Jul 02 21:19:41.774961 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:41.774924    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06a47112-ec12-4baa-bd99-91b2d599254b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvf5d\" (UID: \"06a47112-ec12-4baa-bd99-91b2d599254b\") " pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:41.775246 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:41.775125    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:41.775246 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:41.775218    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06a47112-ec12-4baa-bd99-91b2d599254b-insights-runtime-extractor-tls podName:06a47112-ec12-4baa-bd99-91b2d599254b nodeName:}" failed. No retries permitted until 2026-07-02 21:19:45.775196794 +0000 UTC m=+57.401698651 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/06a47112-ec12-4baa-bd99-91b2d599254b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pvf5d" (UID: "06a47112-ec12-4baa-bd99-91b2d599254b") : secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:41.988246 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:41.988203    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:42.084109 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:42.084077    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:42.550074 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:42.550046    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4xxrj_0c7a0d2a-4e86-46aa-9190-a0ce20694c85/serve-healthcheck-canary/0.log"
Jul 02 21:19:43.255180 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:43.255144    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-8gmq5" event={"ID":"ee079266-209e-4bed-9054-946392b449c3","Type":"ContainerStarted","Data":"064a137b7d1468ebe443d6aa439f4118a243996345b59477e16019ad9f78ea45"}
Jul 02 21:19:43.272035 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:43.271988    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-8gmq5" podStartSLOduration=1.400106435 podStartE2EDuration="3.271975151s" totalImagesPullingTime="1.871868716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:43.270743745 +0000 UTC m=+54.897245648" watchObservedRunningTime="2026-07-02 21:19:43.271975151 +0000 UTC m=+54.898477026"
Jul 02 21:19:43.349730 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:43.349705    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-pbcv9_78977169-1717-4b34-b8ac-7f4562f6be64/kube-storage-version-migrator-operator/0.log"
Jul 02 21:19:44.035453 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:44.035415    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:44.140416 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:44.140379    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:44.237079 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:44.237044    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:44.336369 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:44.336295    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:45.071983 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:45.071952    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:45.490941 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:45.490910    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:45.687688 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:45.687661    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:45.810323 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:45.810256    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06a47112-ec12-4baa-bd99-91b2d599254b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvf5d\" (UID: \"06a47112-ec12-4baa-bd99-91b2d599254b\") " pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:45.810516 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:45.810392    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:45.810516 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:19:45.810448    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06a47112-ec12-4baa-bd99-91b2d599254b-insights-runtime-extractor-tls podName:06a47112-ec12-4baa-bd99-91b2d599254b nodeName:}" failed. No retries permitted until 2026-07-02 21:19:53.810435087 +0000 UTC m=+65.436936941 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/06a47112-ec12-4baa-bd99-91b2d599254b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pvf5d" (UID: "06a47112-ec12-4baa-bd99-91b2d599254b") : secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:45.895102 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:45.895066    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:46.088124 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:46.088047    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:46.290269 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:46.290242    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:47.497351 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:47.497318    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:47.595508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:47.595456    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:47.696216 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:47.696182    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:47.756277 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:47.756208    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:47.901747 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:47.901714    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:48.098155 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:48.098079    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:48.283998 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:48.283968    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:49.086542 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:49.086511    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:49.378679 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:49.378598    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:49.889619 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:49.889585    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:50.188680 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:50.188657    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2725k"
Jul 02 21:19:50.315504 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:50.315474    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:50.574095 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:50.574019    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:50.702317 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:50.702286    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:51.089109 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:51.089082    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:51.295650 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:51.295624    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:51.677274 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:51.677242    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:52.094286 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:52.094206    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:52.686582 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:52.686552    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:52.786528 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:52.786500    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:53.111145 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:53.111076    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:53.483138 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:53.483111    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:53.691625 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:53.691596    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:53.872878 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:53.872806    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06a47112-ec12-4baa-bd99-91b2d599254b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvf5d\" (UID: \"06a47112-ec12-4baa-bd99-91b2d599254b\") " pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:53.875066 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:53.875049    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06a47112-ec12-4baa-bd99-91b2d599254b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvf5d\" (UID: \"06a47112-ec12-4baa-bd99-91b2d599254b\") " pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:53.903417 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:53.903388    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m4mjq\""
Jul 02 21:19:53.910978 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:53.910960    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pvf5d"
Jul 02 21:19:54.033518 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:54.033494    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pvf5d"]
Jul 02 21:19:54.036196 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:19:54.036150    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a47112_ec12_4baa_bd99_91b2d599254b.slice/crio-7152460aa5dbea272dd2da337d763b018b7710c7b0a68760c220f0a30eb3a5df WatchSource:0}: Error finding container 7152460aa5dbea272dd2da337d763b018b7710c7b0a68760c220f0a30eb3a5df: Status 404 returned error can't find the container with id 7152460aa5dbea272dd2da337d763b018b7710c7b0a68760c220f0a30eb3a5df
Jul 02 21:19:54.088112 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:54.088077    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:54.283078 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:54.283011    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pvf5d" event={"ID":"06a47112-ec12-4baa-bd99-91b2d599254b","Type":"ContainerStarted","Data":"7152460aa5dbea272dd2da337d763b018b7710c7b0a68760c220f0a30eb3a5df"}
Jul 02 21:19:54.583347 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:54.583257    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:54.679783 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:54.679720    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret\") pod \"global-pull-secret-syncer-gpccf\" (UID: \"3a6e6de1-8b1b-478e-8dce-996e81502c1e\") " pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:19:54.679989 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:54.679870    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs\") pod \"network-metrics-daemon-pxcm5\" (UID: \"28a7f381-3db6-43e5-9690-d409f048dcf2\") " pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:19:54.683880 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:54.683853    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 21:19:54.685315 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:54.685099    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 21:19:54.694117 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:54.694094    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a6e6de1-8b1b-478e-8dce-996e81502c1e-original-pull-secret\") pod \"global-pull-secret-syncer-gpccf\" (UID: \"3a6e6de1-8b1b-478e-8dce-996e81502c1e\") " pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:19:54.701968 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:54.701917    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28a7f381-3db6-43e5-9690-d409f048dcf2-metrics-certs\") pod \"network-metrics-daemon-pxcm5\" (UID: \"28a7f381-3db6-43e5-9690-d409f048dcf2\") " pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:19:54.719850 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:54.719760    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:54.781342 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:54.781313    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjksx\" (UniqueName: \"kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx\") pod \"network-check-target-b9tf7\" (UID: \"4076e581-5038-4786-9565-c72be1618034\") " pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:19:54.784389 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:54.784332    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 21:19:54.795107 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:54.795085    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 21:19:54.806492 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:54.806449    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjksx\" (UniqueName: \"kubernetes.io/projected/4076e581-5038-4786-9565-c72be1618034-kube-api-access-qjksx\") pod \"network-check-target-b9tf7\" (UID: \"4076e581-5038-4786-9565-c72be1618034\") " pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:19:54.823996 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:54.823790    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zvz49\""
Jul 02 21:19:54.830835 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:54.830627    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zt75x\""
Jul 02 21:19:54.832047 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:54.831849    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxcm5"
Jul 02 21:19:54.834115 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:54.834047    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gpccf"
Jul 02 21:19:54.839352 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:54.839291    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:19:55.031076 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:55.031007    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pxcm5"]
Jul 02 21:19:55.034451 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:19:55.034420    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a7f381_3db6_43e5_9690_d409f048dcf2.slice/crio-048c905fc5330c4f211ba32f6d0025fdb67d5b46b552bae956e41db7c6e9587f WatchSource:0}: Error finding container 048c905fc5330c4f211ba32f6d0025fdb67d5b46b552bae956e41db7c6e9587f: Status 404 returned error can't find the container with id 048c905fc5330c4f211ba32f6d0025fdb67d5b46b552bae956e41db7c6e9587f
Jul 02 21:19:55.042568 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:55.042542    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gpccf"]
Jul 02 21:19:55.045898 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:19:55.045872    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6e6de1_8b1b_478e_8dce_996e81502c1e.slice/crio-9355e8f927280105375c23e025fa4cb28644a9057f6e87b6980f2005fffedfa3 WatchSource:0}: Error finding container 9355e8f927280105375c23e025fa4cb28644a9057f6e87b6980f2005fffedfa3: Status 404 returned error can't find the container with id 9355e8f927280105375c23e025fa4cb28644a9057f6e87b6980f2005fffedfa3
Jul 02 21:19:55.073842 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:55.073812    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b9tf7"]
Jul 02 21:19:55.078492 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:19:55.078469    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4076e581_5038_4786_9565_c72be1618034.slice/crio-1490483d9ca3fb580ef4f5ed4ff14c27fd1e83573c47a66260de0ac80eeeeec7 WatchSource:0}: Error finding container 1490483d9ca3fb580ef4f5ed4ff14c27fd1e83573c47a66260de0ac80eeeeec7: Status 404 returned error can't find the container with id 1490483d9ca3fb580ef4f5ed4ff14c27fd1e83573c47a66260de0ac80eeeeec7
Jul 02 21:19:55.287354 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:55.287313    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gpccf" event={"ID":"3a6e6de1-8b1b-478e-8dce-996e81502c1e","Type":"ContainerStarted","Data":"9355e8f927280105375c23e025fa4cb28644a9057f6e87b6980f2005fffedfa3"}
Jul 02 21:19:55.288618 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:55.288589    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pxcm5" event={"ID":"28a7f381-3db6-43e5-9690-d409f048dcf2","Type":"ContainerStarted","Data":"048c905fc5330c4f211ba32f6d0025fdb67d5b46b552bae956e41db7c6e9587f"}
Jul 02 21:19:55.289813 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:55.289783    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b9tf7" event={"ID":"4076e581-5038-4786-9565-c72be1618034","Type":"ContainerStarted","Data":"1490483d9ca3fb580ef4f5ed4ff14c27fd1e83573c47a66260de0ac80eeeeec7"}
Jul 02 21:19:55.990557 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:55.990435    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:55.998742 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:55.998712    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls\") pod \"image-registry-647f4fc976-pzbv8\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") " pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:56.091153 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:56.091115    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5609d59-a06f-4db8-b564-a0ff92bd58b3-metrics-tls\") pod \"dns-default-7vjmt\" (UID: \"c5609d59-a06f-4db8-b564-a0ff92bd58b3\") " pod="openshift-dns/dns-default-7vjmt"
Jul 02 21:19:56.093742 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:56.093718    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5609d59-a06f-4db8-b564-a0ff92bd58b3-metrics-tls\") pod \"dns-default-7vjmt\" (UID: \"c5609d59-a06f-4db8-b564-a0ff92bd58b3\") " pod="openshift-dns/dns-default-7vjmt"
Jul 02 21:19:56.246423 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:56.246192    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7gtpt\""
Jul 02 21:19:56.254139 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:56.253946    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:19:56.284281 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:56.284257    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qs2b2\""
Jul 02 21:19:56.292707 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:19:56.292684    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7vjmt"
Jul 02 21:20:01.048770 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:01.048713    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7vjmt"]
Jul 02 21:20:01.052627 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:20:01.052598    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5609d59_a06f_4db8_b564_a0ff92bd58b3.slice/crio-07b88c8f34eb9bd9342c4b223875ed604c47d00057728ed8a6c0bae80d6bab60 WatchSource:0}: Error finding container 07b88c8f34eb9bd9342c4b223875ed604c47d00057728ed8a6c0bae80d6bab60: Status 404 returned error can't find the container with id 07b88c8f34eb9bd9342c4b223875ed604c47d00057728ed8a6c0bae80d6bab60
Jul 02 21:20:01.068885 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:01.068829    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-647f4fc976-pzbv8"]
Jul 02 21:20:01.073203 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:20:01.073012    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d8e442e_419a_417c_bb3a_7227bf8e964f.slice/crio-662dc792244fba8dce67f415b65065158615537072ff212ae344c952738b57cf WatchSource:0}: Error finding container 662dc792244fba8dce67f415b65065158615537072ff212ae344c952738b57cf: Status 404 returned error can't find the container with id 662dc792244fba8dce67f415b65065158615537072ff212ae344c952738b57cf
Jul 02 21:20:01.309490 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:01.309455    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gpccf" event={"ID":"3a6e6de1-8b1b-478e-8dce-996e81502c1e","Type":"ContainerStarted","Data":"54c004f102275508e8b7b3d211cff4f55e6c6134029dbc94a12c83bbaefdbe89"}
Jul 02 21:20:01.311306 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:01.311282    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pxcm5" event={"ID":"28a7f381-3db6-43e5-9690-d409f048dcf2","Type":"ContainerStarted","Data":"1a4699054171e55214873b95639511afba6dae826263ba7bb8c0461210729afe"}
Jul 02 21:20:01.311453 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:01.311329    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pxcm5" event={"ID":"28a7f381-3db6-43e5-9690-d409f048dcf2","Type":"ContainerStarted","Data":"a9a0d2ca52d6953de3485e6bb7bd95b8d353be29b21b335afb8a1c9e223859b3"}
Jul 02 21:20:01.312792 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:01.312764    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pvf5d" event={"ID":"06a47112-ec12-4baa-bd99-91b2d599254b","Type":"ContainerStarted","Data":"07f1c2eb6dd99002c01224c1b3b0cf060ccc600a132e33d50c50ea3c1181389b"}
Jul 02 21:20:01.314314 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:01.314292    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b9tf7" event={"ID":"4076e581-5038-4786-9565-c72be1618034","Type":"ContainerStarted","Data":"5bbae96b6936e5f02a77ea69e530caed3a0f3ba4d0caae5ecc499a01117b8486"}
Jul 02 21:20:01.314419 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:01.314359    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:20:01.315378 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:01.315361    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7vjmt" event={"ID":"c5609d59-a06f-4db8-b564-a0ff92bd58b3","Type":"ContainerStarted","Data":"07b88c8f34eb9bd9342c4b223875ed604c47d00057728ed8a6c0bae80d6bab60"}
Jul 02 21:20:01.316793 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:01.316775    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-647f4fc976-pzbv8" event={"ID":"8d8e442e-419a-417c-bb3a-7227bf8e964f","Type":"ContainerStarted","Data":"d26ae13f5fe24eb16751d40424657e34b6843a66993687488be5f5647291ea02"}
Jul 02 21:20:01.316793 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:01.316804    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-647f4fc976-pzbv8" event={"ID":"8d8e442e-419a-417c-bb3a-7227bf8e964f","Type":"ContainerStarted","Data":"662dc792244fba8dce67f415b65065158615537072ff212ae344c952738b57cf"}
Jul 02 21:20:01.316959 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:01.316906    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:20:01.328383 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:01.328344    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gpccf" podStartSLOduration=66.477974382 podStartE2EDuration="1m12.328332014s" totalImagesPullingTime="5.850357632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:01.326930408 +0000 UTC m=+72.953432284" watchObservedRunningTime="2026-07-02 21:20:01.328332014 +0000 UTC m=+72.954833867"
Jul 02 21:20:01.352108 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:01.352033    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pxcm5" podStartSLOduration=66.51165849 podStartE2EDuration="1m12.352016026s" totalImagesPullingTime="5.840357536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:01.351083187 +0000 UTC m=+72.977585093" watchObservedRunningTime="2026-07-02 21:20:01.352016026 +0000 UTC m=+72.978517899"
Jul 02 21:20:01.378963 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:01.378901    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-647f4fc976-pzbv8" podStartSLOduration=72.378879144 podStartE2EDuration="1m12.378879144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:01.377084547 +0000 UTC m=+73.003586460" watchObservedRunningTime="2026-07-02 21:20:01.378879144 +0000 UTC m=+73.005381022"
Jul 02 21:20:01.402304 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:01.402257    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b9tf7" podStartSLOduration=66.605404439 podStartE2EDuration="1m12.402242106s" totalImagesPullingTime="5.796837667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:01.401839245 +0000 UTC m=+73.028341139" watchObservedRunningTime="2026-07-02 21:20:01.402242106 +0000 UTC m=+73.028743982"
Jul 02 21:20:02.327550 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:02.327503    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pvf5d" event={"ID":"06a47112-ec12-4baa-bd99-91b2d599254b","Type":"ContainerStarted","Data":"8d27a013960f3dee61593d1852f0027526bc84cc6bc3c68a7521530a2fcda8f9"}
Jul 02 21:20:03.143351 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.143250    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f8dd4878-hb96w"]
Jul 02 21:20:03.146938 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.146634    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f8dd4878-hb96w"
Jul 02 21:20:03.160125 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.160104    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 02 21:20:03.160125 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.160114    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 02 21:20:03.161300 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.161277    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 02 21:20:03.161421 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.161310    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 02 21:20:03.180835 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.180808    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:03.197015 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.196992    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f8dd4878-hb96w"]
Jul 02 21:20:03.250692 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.250654    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/57f57af5-d982-418d-86ab-b9431076f1dc-klusterlet-config\") pod \"klusterlet-addon-workmgr-76f8dd4878-hb96w\" (UID: \"57f57af5-d982-418d-86ab-b9431076f1dc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f8dd4878-hb96w"
Jul 02 21:20:03.250862 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.250703    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc64d\" (UniqueName: \"kubernetes.io/projected/57f57af5-d982-418d-86ab-b9431076f1dc-kube-api-access-zc64d\") pod \"klusterlet-addon-workmgr-76f8dd4878-hb96w\" (UID: \"57f57af5-d982-418d-86ab-b9431076f1dc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f8dd4878-hb96w"
Jul 02 21:20:03.250862 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.250736    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57f57af5-d982-418d-86ab-b9431076f1dc-tmp\") pod \"klusterlet-addon-workmgr-76f8dd4878-hb96w\" (UID: \"57f57af5-d982-418d-86ab-b9431076f1dc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f8dd4878-hb96w"
Jul 02 21:20:03.254640 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.254611    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-ld997"]
Jul 02 21:20:03.257874 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.257854    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ld997"
Jul 02 21:20:03.270764 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.270741    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 02 21:20:03.270881 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.270800    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-j2br7\""
Jul 02 21:20:03.270881 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.270802    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 02 21:20:03.288837 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.288815    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-ld997"]
Jul 02 21:20:03.351716 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.351686    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/48ccaac6-b95b-4f34-8fc1-be27faf2b2fc-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-ld997\" (UID: \"48ccaac6-b95b-4f34-8fc1-be27faf2b2fc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ld997"
Jul 02 21:20:03.352128 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.351764    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48ccaac6-b95b-4f34-8fc1-be27faf2b2fc-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ld997\" (UID: \"48ccaac6-b95b-4f34-8fc1-be27faf2b2fc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ld997"
Jul 02 21:20:03.352128 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.351798    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/57f57af5-d982-418d-86ab-b9431076f1dc-klusterlet-config\") pod \"klusterlet-addon-workmgr-76f8dd4878-hb96w\" (UID: \"57f57af5-d982-418d-86ab-b9431076f1dc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f8dd4878-hb96w"
Jul 02 21:20:03.352128 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.351833    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc64d\" (UniqueName: \"kubernetes.io/projected/57f57af5-d982-418d-86ab-b9431076f1dc-kube-api-access-zc64d\") pod \"klusterlet-addon-workmgr-76f8dd4878-hb96w\" (UID: \"57f57af5-d982-418d-86ab-b9431076f1dc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f8dd4878-hb96w"
Jul 02 21:20:03.352128 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.351861    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57f57af5-d982-418d-86ab-b9431076f1dc-tmp\") pod \"klusterlet-addon-workmgr-76f8dd4878-hb96w\" (UID: \"57f57af5-d982-418d-86ab-b9431076f1dc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f8dd4878-hb96w"
Jul 02 21:20:03.352317 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.352253    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57f57af5-d982-418d-86ab-b9431076f1dc-tmp\") pod \"klusterlet-addon-workmgr-76f8dd4878-hb96w\" (UID: \"57f57af5-d982-418d-86ab-b9431076f1dc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f8dd4878-hb96w"
Jul 02 21:20:03.354493 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.354466    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/57f57af5-d982-418d-86ab-b9431076f1dc-klusterlet-config\") pod \"klusterlet-addon-workmgr-76f8dd4878-hb96w\" (UID: \"57f57af5-d982-418d-86ab-b9431076f1dc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f8dd4878-hb96w"
Jul 02 21:20:03.421157 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.421088    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:03.439146 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.439119    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc64d\" (UniqueName: \"kubernetes.io/projected/57f57af5-d982-418d-86ab-b9431076f1dc-kube-api-access-zc64d\") pod \"klusterlet-addon-workmgr-76f8dd4878-hb96w\" (UID: \"57f57af5-d982-418d-86ab-b9431076f1dc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f8dd4878-hb96w"
Jul 02 21:20:03.453044 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.453017    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48ccaac6-b95b-4f34-8fc1-be27faf2b2fc-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ld997\" (UID: \"48ccaac6-b95b-4f34-8fc1-be27faf2b2fc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ld997"
Jul 02 21:20:03.453191 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.453076    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/48ccaac6-b95b-4f34-8fc1-be27faf2b2fc-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-ld997\" (UID: \"48ccaac6-b95b-4f34-8fc1-be27faf2b2fc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ld997"
Jul 02 21:20:03.453737 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.453717    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/48ccaac6-b95b-4f34-8fc1-be27faf2b2fc-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-ld997\" (UID: \"48ccaac6-b95b-4f34-8fc1-be27faf2b2fc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ld997"
Jul 02 21:20:03.455477 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.455456    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48ccaac6-b95b-4f34-8fc1-be27faf2b2fc-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ld997\" (UID: \"48ccaac6-b95b-4f34-8fc1-be27faf2b2fc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ld997"
Jul 02 21:20:03.459345 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.459321    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f8dd4878-hb96w"
Jul 02 21:20:03.567844 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.567814    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ld997"
Jul 02 21:20:03.663318 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.663279    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:03.902084 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:03.902044    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:04.185717 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:04.185629    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:04.329838 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:04.329808    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:04.474874 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:04.474815    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:04.567275 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:04.567247    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-ld997"]
Jul 02 21:20:04.579918 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:20:04.579889    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ccaac6_b95b_4f34_8fc1_be27faf2b2fc.slice/crio-fad7d6d975e781b1d59d83845b83f5d64fc1e3081a96c324b912bbd4b4791ebc WatchSource:0}: Error finding container fad7d6d975e781b1d59d83845b83f5d64fc1e3081a96c324b912bbd4b4791ebc: Status 404 returned error can't find the container with id fad7d6d975e781b1d59d83845b83f5d64fc1e3081a96c324b912bbd4b4791ebc
Jul 02 21:20:04.776054 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:04.775964    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f8dd4878-hb96w"]
Jul 02 21:20:04.779998 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:20:04.779975    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f57af5_d982_418d_86ab_b9431076f1dc.slice/crio-de153609940eb8515b3a6258ecb37e649a70c8bf7792239fb31004eaf44d6371 WatchSource:0}: Error finding container de153609940eb8515b3a6258ecb37e649a70c8bf7792239fb31004eaf44d6371: Status 404 returned error can't find the container with id de153609940eb8515b3a6258ecb37e649a70c8bf7792239fb31004eaf44d6371
Jul 02 21:20:04.812355 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:04.812321    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:05.160837 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:05.160806    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:05.258665 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:05.258631    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:05.342550 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:05.342157    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pvf5d" event={"ID":"06a47112-ec12-4baa-bd99-91b2d599254b","Type":"ContainerStarted","Data":"9dbc2a9a9026e4f42ef88d53d84dd8c8c2327d1b78867e6d94593bdc08b8de02"}
Jul 02 21:20:05.345891 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:05.345294    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7vjmt" event={"ID":"c5609d59-a06f-4db8-b564-a0ff92bd58b3","Type":"ContainerStarted","Data":"3b2a2d011b421639426a4a866f7e3a0c6a30a7870740865df7c863ef26694909"}
Jul 02 21:20:05.345891 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:05.345341    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7vjmt" event={"ID":"c5609d59-a06f-4db8-b564-a0ff92bd58b3","Type":"ContainerStarted","Data":"168c61baaa7dd3b3da0da4d7f6abeffd7f9ef3e3f8a3bb661af2bc0063aea2d2"}
Jul 02 21:20:05.345891 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:05.345452    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7vjmt"
Jul 02 21:20:05.347719 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:05.347660    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ld997" event={"ID":"48ccaac6-b95b-4f34-8fc1-be27faf2b2fc","Type":"ContainerStarted","Data":"fad7d6d975e781b1d59d83845b83f5d64fc1e3081a96c324b912bbd4b4791ebc"}
Jul 02 21:20:05.349492 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:05.349465    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f8dd4878-hb96w" event={"ID":"57f57af5-d982-418d-86ab-b9431076f1dc","Type":"ContainerStarted","Data":"de153609940eb8515b3a6258ecb37e649a70c8bf7792239fb31004eaf44d6371"}
Jul 02 21:20:05.365798 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:05.365575    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pvf5d" podStartSLOduration=18.158753309 podStartE2EDuration="28.365557611s" totalImagesPullingTime="10.206804302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:19:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:05.364705698 +0000 UTC m=+76.991207574" watchObservedRunningTime="2026-07-02 21:20:05.365557611 +0000 UTC m=+76.992059490"
Jul 02 21:20:05.391260 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:05.390463    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7vjmt" podStartSLOduration=38.061614349 podStartE2EDuration="41.390441565s" totalImagesPullingTime="3.328827216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:05.389602246 +0000 UTC m=+77.016104124" watchObservedRunningTime="2026-07-02 21:20:05.390441565 +0000 UTC m=+77.016943442"
Jul 02 21:20:06.355535 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:06.355496    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ld997" event={"ID":"48ccaac6-b95b-4f34-8fc1-be27faf2b2fc","Type":"ContainerStarted","Data":"daf7860951b3c62a285c0e22fb1a59731a6fba37369d2b678c272cecb30e0629"}
Jul 02 21:20:06.414524 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:06.414462    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ld997" podStartSLOduration=2.165653391 podStartE2EDuration="3.414437831s" totalImagesPullingTime="1.24878444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:06.413128423 +0000 UTC m=+78.039630296" watchObservedRunningTime="2026-07-02 21:20:06.414437831 +0000 UTC m=+78.040939685"
Jul 02 21:20:06.445360 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:06.445322    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:06.535662 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:06.535612    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:06.612612 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:06.611938    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9dcr6"]
Jul 02 21:20:06.623608 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:06.622794    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9dcr6"
Jul 02 21:20:06.623608 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:06.623375    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9dcr6"]
Jul 02 21:20:06.626556 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:06.626054    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zpwxz\""
Jul 02 21:20:06.626556 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:06.626328    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 02 21:20:06.631454 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:06.631427    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:06.681051 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:06.680977    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ef4203cc-0ee5-40b2-935c-6d81bf5b5948-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-9dcr6\" (UID: \"ef4203cc-0ee5-40b2-935c-6d81bf5b5948\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9dcr6"
Jul 02 21:20:06.781892 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:06.781639    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ef4203cc-0ee5-40b2-935c-6d81bf5b5948-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-9dcr6\" (UID: \"ef4203cc-0ee5-40b2-935c-6d81bf5b5948\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9dcr6"
Jul 02 21:20:06.781892 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:20:06.781814    2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 02 21:20:06.782408 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:20:06.782156    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef4203cc-0ee5-40b2-935c-6d81bf5b5948-tls-certificates podName:ef4203cc-0ee5-40b2-935c-6d81bf5b5948 nodeName:}" failed. No retries permitted until 2026-07-02 21:20:07.282127394 +0000 UTC m=+78.908629262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/ef4203cc-0ee5-40b2-935c-6d81bf5b5948-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-9dcr6" (UID: "ef4203cc-0ee5-40b2-935c-6d81bf5b5948") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 02 21:20:07.286572 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:07.286535    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ef4203cc-0ee5-40b2-935c-6d81bf5b5948-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-9dcr6\" (UID: \"ef4203cc-0ee5-40b2-935c-6d81bf5b5948\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9dcr6"
Jul 02 21:20:07.286759 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:20:07.286698    2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 02 21:20:07.286819 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:20:07.286768    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef4203cc-0ee5-40b2-935c-6d81bf5b5948-tls-certificates podName:ef4203cc-0ee5-40b2-935c-6d81bf5b5948 nodeName:}" failed. No retries permitted until 2026-07-02 21:20:08.286749855 +0000 UTC m=+79.913251728 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/ef4203cc-0ee5-40b2-935c-6d81bf5b5948-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-9dcr6" (UID: "ef4203cc-0ee5-40b2-935c-6d81bf5b5948") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 02 21:20:07.380200 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:07.380150    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:07.522522 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:07.522478    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:07.902819 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:07.902784    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:08.003188 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:08.003140    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:08.294729 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:08.294705    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ef4203cc-0ee5-40b2-935c-6d81bf5b5948-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-9dcr6\" (UID: \"ef4203cc-0ee5-40b2-935c-6d81bf5b5948\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9dcr6"
Jul 02 21:20:08.296768 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:08.296750    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ef4203cc-0ee5-40b2-935c-6d81bf5b5948-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-9dcr6\" (UID: \"ef4203cc-0ee5-40b2-935c-6d81bf5b5948\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9dcr6"
Jul 02 21:20:08.367828 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:08.367796    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f8dd4878-hb96w" event={"ID":"57f57af5-d982-418d-86ab-b9431076f1dc","Type":"ContainerStarted","Data":"bb13678652c967672f3c89cb3d6ebb31f1606818e1556c1a06a696e10bb96609"}
Jul 02 21:20:08.368024 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:08.367996    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f8dd4878-hb96w"
Jul 02 21:20:08.369273 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:08.369256    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f8dd4878-hb96w"
Jul 02 21:20:08.385689 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:08.385648    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f8dd4878-hb96w" podStartSLOduration=1.9169440309999999 podStartE2EDuration="5.385635231s" totalImagesPullingTime="3.4686912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:08.384065589 +0000 UTC m=+80.010567476" watchObservedRunningTime="2026-07-02 21:20:08.385635231 +0000 UTC m=+80.012137107"
Jul 02 21:20:08.436201 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:08.436164    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9dcr6"
Jul 02 21:20:08.554667 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:08.554642    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9dcr6"]
Jul 02 21:20:08.557772 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:20:08.557749    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4203cc_0ee5_40b2_935c_6d81bf5b5948.slice/crio-498e506d8bc409562fea839dc3cc9e5440c7aaf62309dbee66c2c2bc11d912a4 WatchSource:0}: Error finding container 498e506d8bc409562fea839dc3cc9e5440c7aaf62309dbee66c2c2bc11d912a4: Status 404 returned error can't find the container with id 498e506d8bc409562fea839dc3cc9e5440c7aaf62309dbee66c2c2bc11d912a4
Jul 02 21:20:09.373233 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:09.373198    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9dcr6" event={"ID":"ef4203cc-0ee5-40b2-935c-6d81bf5b5948","Type":"ContainerStarted","Data":"498e506d8bc409562fea839dc3cc9e5440c7aaf62309dbee66c2c2bc11d912a4"}
Jul 02 21:20:10.378344 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:10.378305    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9dcr6" event={"ID":"ef4203cc-0ee5-40b2-935c-6d81bf5b5948","Type":"ContainerStarted","Data":"cdfc957609784373a19b60567bf04861b9bcb7e490720b3d98a8da4c7ce2fafb"}
Jul 02 21:20:10.400940 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:10.400888    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9dcr6" podStartSLOduration=3.382343018 podStartE2EDuration="4.400874015s" totalImagesPullingTime="1.018530997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:10.40052529 +0000 UTC m=+82.027027191" watchObservedRunningTime="2026-07-02 21:20:10.400874015 +0000 UTC m=+82.027375890"
Jul 02 21:20:11.382091 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:11.382063    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9dcr6"
Jul 02 21:20:11.386787 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:11.386763    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9dcr6"
Jul 02 21:20:11.657201 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:11.657145    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:11.746556 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:11.746530    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:11.928087 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:11.928008    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:13.810386 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:13.810354    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:15.357344 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:15.357310    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7vjmt"
Jul 02 21:20:15.845582 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:15.845548    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:15.977317 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:15.977288    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:16.083524 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.083486    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:16.117025 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.116955    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-scqvn"]
Jul 02 21:20:16.122234 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.122215    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-scqvn"
Jul 02 21:20:16.126795 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.126774    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xpf4p\""
Jul 02 21:20:16.126903 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.126796    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 02 21:20:16.127006 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.126785    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 02 21:20:16.128002 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.127985    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 02 21:20:16.128105 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.128050    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 02 21:20:16.129756 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.129740    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 02 21:20:16.135864 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.135843    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-scqvn"]
Jul 02 21:20:16.145037 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.145014    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"]
Jul 02 21:20:16.148489 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.148474    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"
Jul 02 21:20:16.151876 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.151860    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 02 21:20:16.152112 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.152092    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jrhph\""
Jul 02 21:20:16.152216 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.152092    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 02 21:20:16.152279 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.152252    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 02 21:20:16.163977 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.163960    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"]
Jul 02 21:20:16.174900 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.174879    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lpkx5"]
Jul 02 21:20:16.178346 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.178332    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.184475 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.184457    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nqfz6\""
Jul 02 21:20:16.184712 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.184673    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 02 21:20:16.184849 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.184833    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 02 21:20:16.185259 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.185246    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 02 21:20:16.212294 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.212264    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:16.254223 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.254196    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-t6jtm\" (UID: \"cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"
Jul 02 21:20:16.254223 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.254227    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab2d2af2-5cf1-4ef5-ba32-685951b3595b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-scqvn\" (UID: \"ab2d2af2-5cf1-4ef5-ba32-685951b3595b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-scqvn"
Jul 02 21:20:16.254467 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.254247    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab2d2af2-5cf1-4ef5-ba32-685951b3595b-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-scqvn\" (UID: \"ab2d2af2-5cf1-4ef5-ba32-685951b3595b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-scqvn"
Jul 02 21:20:16.254467 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.254266    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-sys\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.254467 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.254310    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab2d2af2-5cf1-4ef5-ba32-685951b3595b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-scqvn\" (UID: \"ab2d2af2-5cf1-4ef5-ba32-685951b3595b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-scqvn"
Jul 02 21:20:16.254467 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.254332    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-node-exporter-wtmp\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.254467 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.254350    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-node-exporter-tls\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.254467 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.254373    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.254467 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.254403    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-t6jtm\" (UID: \"cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"
Jul 02 21:20:16.254467 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.254425    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49pbc\" (UniqueName: \"kubernetes.io/projected/ab2d2af2-5cf1-4ef5-ba32-685951b3595b-kube-api-access-49pbc\") pod \"openshift-state-metrics-54cf96dfcf-scqvn\" (UID: \"ab2d2af2-5cf1-4ef5-ba32-685951b3595b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-scqvn"
Jul 02 21:20:16.254467 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.254441    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-metrics-client-ca\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.254467 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.254457    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-root\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.254983 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.254490    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-t6jtm\" (UID: \"cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"
Jul 02 21:20:16.254983 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.254517    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9gvp\" (UniqueName: \"kubernetes.io/projected/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-kube-api-access-s9gvp\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.254983 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.254547    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-t6jtm\" (UID: \"cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"
Jul 02 21:20:16.254983 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.254571    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-node-exporter-accelerators-collector-config\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.254983 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.254624    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk8pc\" (UniqueName: \"kubernetes.io/projected/cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9-kube-api-access-xk8pc\") pod \"kube-state-metrics-7c9dbb8b7d-t6jtm\" (UID: \"cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"
Jul 02 21:20:16.254983 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.254677    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-node-exporter-textfile\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.254983 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.254710    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-t6jtm\" (UID: \"cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"
Jul 02 21:20:16.258259 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.258227    2575 patch_prober.go:28] interesting pod/image-registry-647f4fc976-pzbv8 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 02 21:20:16.258388 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.258284    2575 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-647f4fc976-pzbv8" podUID="8d8e442e-419a-417c-bb3a-7227bf8e964f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:20:16.310844 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.310814    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:16.355992 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.355955    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49pbc\" (UniqueName: \"kubernetes.io/projected/ab2d2af2-5cf1-4ef5-ba32-685951b3595b-kube-api-access-49pbc\") pod \"openshift-state-metrics-54cf96dfcf-scqvn\" (UID: \"ab2d2af2-5cf1-4ef5-ba32-685951b3595b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-scqvn"
Jul 02 21:20:16.355992 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.355992    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-metrics-client-ca\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.356260 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356012    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-root\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.356260 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356034    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-t6jtm\" (UID: \"cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"
Jul 02 21:20:16.356260 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356060    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9gvp\" (UniqueName: \"kubernetes.io/projected/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-kube-api-access-s9gvp\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.356260 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356092    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-root\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.356260 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356097    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-t6jtm\" (UID: \"cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"
Jul 02 21:20:16.356260 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356147    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-node-exporter-accelerators-collector-config\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.356260 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356211    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk8pc\" (UniqueName: \"kubernetes.io/projected/cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9-kube-api-access-xk8pc\") pod \"kube-state-metrics-7c9dbb8b7d-t6jtm\" (UID: \"cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"
Jul 02 21:20:16.356260 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356245    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-node-exporter-textfile\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.356661 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356276    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-t6jtm\" (UID: \"cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"
Jul 02 21:20:16.356661 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356316    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-t6jtm\" (UID: \"cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"
Jul 02 21:20:16.356661 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356343    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab2d2af2-5cf1-4ef5-ba32-685951b3595b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-scqvn\" (UID: \"ab2d2af2-5cf1-4ef5-ba32-685951b3595b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-scqvn"
Jul 02 21:20:16.356809 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356677    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-metrics-client-ca\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.356809 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356689    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-t6jtm\" (UID: \"cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"
Jul 02 21:20:16.356809 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356729    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab2d2af2-5cf1-4ef5-ba32-685951b3595b-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-scqvn\" (UID: \"ab2d2af2-5cf1-4ef5-ba32-685951b3595b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-scqvn"
Jul 02 21:20:16.356809 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356757    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-node-exporter-textfile\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.356809 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356764    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-sys\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.357053 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356812    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab2d2af2-5cf1-4ef5-ba32-685951b3595b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-scqvn\" (UID: \"ab2d2af2-5cf1-4ef5-ba32-685951b3595b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-scqvn"
Jul 02 21:20:16.357053 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356841    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-node-exporter-wtmp\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.357053 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356870    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-node-exporter-tls\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.357053 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356894    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.357053 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356921    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-t6jtm\" (UID: \"cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"
Jul 02 21:20:16.357053 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356994    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-t6jtm\" (UID: \"cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"
Jul 02 21:20:16.357053 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.356994    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-t6jtm\" (UID: \"cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"
Jul 02 21:20:16.357982 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.357058    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-sys\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.357982 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.357108    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-node-exporter-wtmp\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.357982 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:20:16.357240    2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 02 21:20:16.357982 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:20:16.357344    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-node-exporter-tls podName:a1f133dc-484c-4d5a-a67e-9fd63dbf861a nodeName:}" failed. No retries permitted until 2026-07-02 21:20:16.857324366 +0000 UTC m=+88.483826231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-node-exporter-tls") pod "node-exporter-lpkx5" (UID: "a1f133dc-484c-4d5a-a67e-9fd63dbf861a") : secret "node-exporter-tls" not found
Jul 02 21:20:16.357982 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.357484    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab2d2af2-5cf1-4ef5-ba32-685951b3595b-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-scqvn\" (UID: \"ab2d2af2-5cf1-4ef5-ba32-685951b3595b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-scqvn"
Jul 02 21:20:16.358740 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.358611    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-node-exporter-accelerators-collector-config\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.359315 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.359292    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-t6jtm\" (UID: \"cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"
Jul 02 21:20:16.359417 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.359334    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab2d2af2-5cf1-4ef5-ba32-685951b3595b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-scqvn\" (UID: \"ab2d2af2-5cf1-4ef5-ba32-685951b3595b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-scqvn"
Jul 02 21:20:16.359657 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.359637    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-t6jtm\" (UID: \"cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"
Jul 02 21:20:16.359724 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.359644    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab2d2af2-5cf1-4ef5-ba32-685951b3595b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-scqvn\" (UID: \"ab2d2af2-5cf1-4ef5-ba32-685951b3595b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-scqvn"
Jul 02 21:20:16.359775 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.359751    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.366206 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.366161    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49pbc\" (UniqueName: \"kubernetes.io/projected/ab2d2af2-5cf1-4ef5-ba32-685951b3595b-kube-api-access-49pbc\") pod \"openshift-state-metrics-54cf96dfcf-scqvn\" (UID: \"ab2d2af2-5cf1-4ef5-ba32-685951b3595b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-scqvn"
Jul 02 21:20:16.366722 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.366690    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk8pc\" (UniqueName: \"kubernetes.io/projected/cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9-kube-api-access-xk8pc\") pod \"kube-state-metrics-7c9dbb8b7d-t6jtm\" (UID: \"cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"
Jul 02 21:20:16.369228 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.369155    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9gvp\" (UniqueName: \"kubernetes.io/projected/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-kube-api-access-s9gvp\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.411975 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.411946    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:16.431905 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.431880    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-scqvn"
Jul 02 21:20:16.457879 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.457847    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"
Jul 02 21:20:16.511010 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.510977    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:16.583797 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.583697    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-scqvn"]
Jul 02 21:20:16.586999 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:20:16.586971    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab2d2af2_5cf1_4ef5_ba32_685951b3595b.slice/crio-5cf0f407ca5d8a32a7fcc203546c6d86d31454d3bac2a4b9681d29575657b208 WatchSource:0}: Error finding container 5cf0f407ca5d8a32a7fcc203546c6d86d31454d3bac2a4b9681d29575657b208: Status 404 returned error can't find the container with id 5cf0f407ca5d8a32a7fcc203546c6d86d31454d3bac2a4b9681d29575657b208
Jul 02 21:20:16.604420 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.604367    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm"]
Jul 02 21:20:16.607002 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:20:16.606977    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8858d5_77d8_4ad7_b5d3_f8bc32332bc9.slice/crio-6d64cf1e4f349ef122242a737885c3964abcb114910b745482c5f3bdc5d9a75e WatchSource:0}: Error finding container 6d64cf1e4f349ef122242a737885c3964abcb114910b745482c5f3bdc5d9a75e: Status 404 returned error can't find the container with id 6d64cf1e4f349ef122242a737885c3964abcb114910b745482c5f3bdc5d9a75e
Jul 02 21:20:16.607326 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.607303    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:16.680254 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.680231    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:16.861482 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.861453    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-node-exporter-tls\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:16.863676 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:16.863656    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1f133dc-484c-4d5a-a67e-9fd63dbf861a-node-exporter-tls\") pod \"node-exporter-lpkx5\" (UID: \"a1f133dc-484c-4d5a-a67e-9fd63dbf861a\") " pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:17.087487 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:17.087445    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lpkx5"
Jul 02 21:20:17.102406 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:17.102379    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:17.106232 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:20:17.106206    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f133dc_484c_4d5a_a67e_9fd63dbf861a.slice/crio-7b72ac64ffa81e3f7817bc1b9dd85083ca1b8beed418f7a69ab19c89d7b68de6 WatchSource:0}: Error finding container 7b72ac64ffa81e3f7817bc1b9dd85083ca1b8beed418f7a69ab19c89d7b68de6: Status 404 returned error can't find the container with id 7b72ac64ffa81e3f7817bc1b9dd85083ca1b8beed418f7a69ab19c89d7b68de6
Jul 02 21:20:17.202283 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:17.202215    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:17.311358 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:17.311317    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:17.404606 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:17.404568    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm" event={"ID":"cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9","Type":"ContainerStarted","Data":"6d64cf1e4f349ef122242a737885c3964abcb114910b745482c5f3bdc5d9a75e"}
Jul 02 21:20:17.406911 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:17.406878    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-scqvn" event={"ID":"ab2d2af2-5cf1-4ef5-ba32-685951b3595b","Type":"ContainerStarted","Data":"e5a51a5534f9dc779a1763de2a91611e490ce0ed9f2ff56bdb6007eeb17147fc"}
Jul 02 21:20:17.407045 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:17.406941    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-scqvn" event={"ID":"ab2d2af2-5cf1-4ef5-ba32-685951b3595b","Type":"ContainerStarted","Data":"ba01529117c93a7df2df56b0fdf1c097716bef097ae0a00028c9ec289614194a"}
Jul 02 21:20:17.407045 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:17.406972    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-scqvn" event={"ID":"ab2d2af2-5cf1-4ef5-ba32-685951b3595b","Type":"ContainerStarted","Data":"5cf0f407ca5d8a32a7fcc203546c6d86d31454d3bac2a4b9681d29575657b208"}
Jul 02 21:20:17.408402 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:17.408375    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lpkx5" event={"ID":"a1f133dc-484c-4d5a-a67e-9fd63dbf861a","Type":"ContainerStarted","Data":"7b72ac64ffa81e3f7817bc1b9dd85083ca1b8beed418f7a69ab19c89d7b68de6"}
Jul 02 21:20:18.415693 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:18.415661    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm" event={"ID":"cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9","Type":"ContainerStarted","Data":"bae45aaf3555321a4cef77c45a1c92f750c2fd15b67d811a40be24089ff563b5"}
Jul 02 21:20:18.416127 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:18.415729    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm" event={"ID":"cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9","Type":"ContainerStarted","Data":"610a346abfc018ebcc6daee086891e77ca9823bd6704184be8c576d90a24a524"}
Jul 02 21:20:18.416127 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:18.415757    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm" event={"ID":"cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9","Type":"ContainerStarted","Data":"804f8b673e15de330155cd649c325cad929721602d36c1766edce20f59f4a0f3"}
Jul 02 21:20:18.418674 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:18.418639    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-scqvn" event={"ID":"ab2d2af2-5cf1-4ef5-ba32-685951b3595b","Type":"ContainerStarted","Data":"875659a0558c0963e412eace4bf459752d98b774e68daf83d30c8d0c70f04ffc"}
Jul 02 21:20:18.439454 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:18.439386    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-t6jtm" podStartSLOduration=1.121479695 podStartE2EDuration="2.439367189s" totalImagesPullingTime="1.317887494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:18.438110646 +0000 UTC m=+90.064612550" watchObservedRunningTime="2026-07-02 21:20:18.439367189 +0000 UTC m=+90.065869084"
Jul 02 21:20:18.465062 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:18.465003    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-scqvn" podStartSLOduration=1.248223044 podStartE2EDuration="2.464980127s" totalImagesPullingTime="1.216757083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:18.463481117 +0000 UTC m=+90.089982993" watchObservedRunningTime="2026-07-02 21:20:18.464980127 +0000 UTC m=+90.091482004"
Jul 02 21:20:19.062076 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.062041    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:19.126344 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.126308    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"]
Jul 02 21:20:19.130377 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.130350    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.134909 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.134887    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 02 21:20:19.135035 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.134974    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 02 21:20:19.135095 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.135032    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-d4kh4\""
Jul 02 21:20:19.135095 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.135077    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 02 21:20:19.135210 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.135133    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4auiqd6ufqrmt\""
Jul 02 21:20:19.135348 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.135332    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 02 21:20:19.135410 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.135367    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 02 21:20:19.151447 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.151420    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"]
Jul 02 21:20:19.164872 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.164850    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:19.279996 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.279969    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f1ca83f6-4c9b-4317-8455-239386da3440-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.280124 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.280003    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1ca83f6-4c9b-4317-8455-239386da3440-metrics-client-ca\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.280124 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.280025    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqpq5\" (UniqueName: \"kubernetes.io/projected/f1ca83f6-4c9b-4317-8455-239386da3440-kube-api-access-nqpq5\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.280259 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.280119    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1ca83f6-4c9b-4317-8455-239386da3440-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.280259 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.280158    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1ca83f6-4c9b-4317-8455-239386da3440-secret-grpc-tls\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.280259 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.280212    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f1ca83f6-4c9b-4317-8455-239386da3440-secret-thanos-querier-tls\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.280405 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.280269    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1ca83f6-4c9b-4317-8455-239386da3440-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.280405 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.280326    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f1ca83f6-4c9b-4317-8455-239386da3440-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.380903 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.380840    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f1ca83f6-4c9b-4317-8455-239386da3440-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.380903 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.380878    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f1ca83f6-4c9b-4317-8455-239386da3440-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.381065 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.380909    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1ca83f6-4c9b-4317-8455-239386da3440-metrics-client-ca\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.381065 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.380968    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqpq5\" (UniqueName: \"kubernetes.io/projected/f1ca83f6-4c9b-4317-8455-239386da3440-kube-api-access-nqpq5\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.381065 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.381028    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1ca83f6-4c9b-4317-8455-239386da3440-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.381065 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.381054    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1ca83f6-4c9b-4317-8455-239386da3440-secret-grpc-tls\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.381280 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.381090    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f1ca83f6-4c9b-4317-8455-239386da3440-secret-thanos-querier-tls\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.381280 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.381133    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1ca83f6-4c9b-4317-8455-239386da3440-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.381767 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.381657    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1ca83f6-4c9b-4317-8455-239386da3440-metrics-client-ca\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.384372 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.384149    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1ca83f6-4c9b-4317-8455-239386da3440-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.384372 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.384332    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f1ca83f6-4c9b-4317-8455-239386da3440-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.384372 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.384335    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f1ca83f6-4c9b-4317-8455-239386da3440-secret-thanos-querier-tls\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.384563 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.384541    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1ca83f6-4c9b-4317-8455-239386da3440-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.384639 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.384615    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f1ca83f6-4c9b-4317-8455-239386da3440-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.385585 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.385561    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1ca83f6-4c9b-4317-8455-239386da3440-secret-grpc-tls\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.391002 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.390978    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqpq5\" (UniqueName: \"kubernetes.io/projected/f1ca83f6-4c9b-4317-8455-239386da3440-kube-api-access-nqpq5\") pod \"thanos-querier-bb85b44fd-l7bk2\" (UID: \"f1ca83f6-4c9b-4317-8455-239386da3440\") " pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.423942 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.423914    2575 generic.go:356] "Generic (PLEG): container finished" podID="a1f133dc-484c-4d5a-a67e-9fd63dbf861a" containerID="453328fcbcfe57cc7288124b7199c4a01714b56f268a9c5418740825bb98b829" exitCode=0
Jul 02 21:20:19.424252 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.424002    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lpkx5" event={"ID":"a1f133dc-484c-4d5a-a67e-9fd63dbf861a","Type":"ContainerDied","Data":"453328fcbcfe57cc7288124b7199c4a01714b56f268a9c5418740825bb98b829"}
Jul 02 21:20:19.441036 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.441011    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:19.585961 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.585935    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"]
Jul 02 21:20:19.589114 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:20:19.589089    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ca83f6_4c9b_4317_8455_239386da3440.slice/crio-c5fadaa06e1b259788f94f39cc5744ec90acdeb6829447c1158e6cfe8c533814 WatchSource:0}: Error finding container c5fadaa06e1b259788f94f39cc5744ec90acdeb6829447c1158e6cfe8c533814: Status 404 returned error can't find the container with id c5fadaa06e1b259788f94f39cc5744ec90acdeb6829447c1158e6cfe8c533814
Jul 02 21:20:19.685863 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.685834    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:19.926357 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:19.926324    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:20.431651 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.431608    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2" event={"ID":"f1ca83f6-4c9b-4317-8455-239386da3440","Type":"ContainerStarted","Data":"c5fadaa06e1b259788f94f39cc5744ec90acdeb6829447c1158e6cfe8c533814"}
Jul 02 21:20:20.433886 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.433859    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lpkx5" event={"ID":"a1f133dc-484c-4d5a-a67e-9fd63dbf861a","Type":"ContainerStarted","Data":"b8d260de5084261b11813f98892d07088146b454df3c3866a2547f7c1d480275"}
Jul 02 21:20:20.434006 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.433910    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lpkx5" event={"ID":"a1f133dc-484c-4d5a-a67e-9fd63dbf861a","Type":"ContainerStarted","Data":"7e72539499da734a9d9d6d0a140dc03da06f8e40bff8d1465d890ebe8378dc60"}
Jul 02 21:20:20.583249 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.583200    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lpkx5" podStartSLOduration=3.113171005 podStartE2EDuration="4.583163429s" totalImagesPullingTime="1.403022632s" totalInitContainerRuntime="66.969792ms" isStatefulPod=true podCreationTimestamp="2026-07-02 21:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:20.471748943 +0000 UTC m=+92.098250837" watchObservedRunningTime="2026-07-02 21:20:20.583163429 +0000 UTC m=+92.209665308"
Jul 02 21:20:20.584412 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.584384    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-88d98b7c6-w4njm"]
Jul 02 21:20:20.587632 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.587605    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.591898 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.591852    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5k93oo2mathpd\""
Jul 02 21:20:20.592137 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.592120    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 02 21:20:20.592340 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.592317    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kl2j6\""
Jul 02 21:20:20.592444 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.592358    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 02 21:20:20.592444 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.592372    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 02 21:20:20.592444 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.592427    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 02 21:20:20.608333 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.608305    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-88d98b7c6-w4njm"]
Jul 02 21:20:20.692923 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.692847    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0ca3427a-2e3a-496a-891c-326ea3e54437-audit-log\") pod \"metrics-server-88d98b7c6-w4njm\" (UID: \"0ca3427a-2e3a-496a-891c-326ea3e54437\") " pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.692923 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.692888    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0ca3427a-2e3a-496a-891c-326ea3e54437-secret-metrics-server-client-certs\") pod \"metrics-server-88d98b7c6-w4njm\" (UID: \"0ca3427a-2e3a-496a-891c-326ea3e54437\") " pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.692923 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.692917    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ca3427a-2e3a-496a-891c-326ea3e54437-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-88d98b7c6-w4njm\" (UID: \"0ca3427a-2e3a-496a-891c-326ea3e54437\") " pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.693132 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.692943    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0ca3427a-2e3a-496a-891c-326ea3e54437-metrics-server-audit-profiles\") pod \"metrics-server-88d98b7c6-w4njm\" (UID: \"0ca3427a-2e3a-496a-891c-326ea3e54437\") " pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.693132 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.692971    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0ca3427a-2e3a-496a-891c-326ea3e54437-secret-metrics-server-tls\") pod \"metrics-server-88d98b7c6-w4njm\" (UID: \"0ca3427a-2e3a-496a-891c-326ea3e54437\") " pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.693132 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.692994    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n27hj\" (UniqueName: \"kubernetes.io/projected/0ca3427a-2e3a-496a-891c-326ea3e54437-kube-api-access-n27hj\") pod \"metrics-server-88d98b7c6-w4njm\" (UID: \"0ca3427a-2e3a-496a-891c-326ea3e54437\") " pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.693132 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.693081    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0ca3427a-2e3a-496a-891c-326ea3e54437-client-ca-bundle\") pod \"metrics-server-88d98b7c6-w4njm\" (UID: \"0ca3427a-2e3a-496a-891c-326ea3e54437\") " pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.793659 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.793631    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0ca3427a-2e3a-496a-891c-326ea3e54437-audit-log\") pod \"metrics-server-88d98b7c6-w4njm\" (UID: \"0ca3427a-2e3a-496a-891c-326ea3e54437\") " pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.793659 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.793664    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0ca3427a-2e3a-496a-891c-326ea3e54437-secret-metrics-server-client-certs\") pod \"metrics-server-88d98b7c6-w4njm\" (UID: \"0ca3427a-2e3a-496a-891c-326ea3e54437\") " pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.793872 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.793687    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ca3427a-2e3a-496a-891c-326ea3e54437-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-88d98b7c6-w4njm\" (UID: \"0ca3427a-2e3a-496a-891c-326ea3e54437\") " pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.793872 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.793708    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0ca3427a-2e3a-496a-891c-326ea3e54437-metrics-server-audit-profiles\") pod \"metrics-server-88d98b7c6-w4njm\" (UID: \"0ca3427a-2e3a-496a-891c-326ea3e54437\") " pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.793872 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.793725    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0ca3427a-2e3a-496a-891c-326ea3e54437-secret-metrics-server-tls\") pod \"metrics-server-88d98b7c6-w4njm\" (UID: \"0ca3427a-2e3a-496a-891c-326ea3e54437\") " pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.794006 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.793910    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n27hj\" (UniqueName: \"kubernetes.io/projected/0ca3427a-2e3a-496a-891c-326ea3e54437-kube-api-access-n27hj\") pod \"metrics-server-88d98b7c6-w4njm\" (UID: \"0ca3427a-2e3a-496a-891c-326ea3e54437\") " pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.794006 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.793998    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0ca3427a-2e3a-496a-891c-326ea3e54437-client-ca-bundle\") pod \"metrics-server-88d98b7c6-w4njm\" (UID: \"0ca3427a-2e3a-496a-891c-326ea3e54437\") " pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.794108 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.794069    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0ca3427a-2e3a-496a-891c-326ea3e54437-audit-log\") pod \"metrics-server-88d98b7c6-w4njm\" (UID: \"0ca3427a-2e3a-496a-891c-326ea3e54437\") " pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.794584 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.794559    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ca3427a-2e3a-496a-891c-326ea3e54437-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-88d98b7c6-w4njm\" (UID: \"0ca3427a-2e3a-496a-891c-326ea3e54437\") " pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.794931 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.794908    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0ca3427a-2e3a-496a-891c-326ea3e54437-metrics-server-audit-profiles\") pod \"metrics-server-88d98b7c6-w4njm\" (UID: \"0ca3427a-2e3a-496a-891c-326ea3e54437\") " pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.796302 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.796282    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0ca3427a-2e3a-496a-891c-326ea3e54437-secret-metrics-server-tls\") pod \"metrics-server-88d98b7c6-w4njm\" (UID: \"0ca3427a-2e3a-496a-891c-326ea3e54437\") " pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.796405 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.796347    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0ca3427a-2e3a-496a-891c-326ea3e54437-secret-metrics-server-client-certs\") pod \"metrics-server-88d98b7c6-w4njm\" (UID: \"0ca3427a-2e3a-496a-891c-326ea3e54437\") " pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.796468 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.796447    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0ca3427a-2e3a-496a-891c-326ea3e54437-client-ca-bundle\") pod \"metrics-server-88d98b7c6-w4njm\" (UID: \"0ca3427a-2e3a-496a-891c-326ea3e54437\") " pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.821199 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.821159    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n27hj\" (UniqueName: \"kubernetes.io/projected/0ca3427a-2e3a-496a-891c-326ea3e54437-kube-api-access-n27hj\") pod \"metrics-server-88d98b7c6-w4njm\" (UID: \"0ca3427a-2e3a-496a-891c-326ea3e54437\") " pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.899012 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.898981    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:20.964494 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:20.964420    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:21.479485 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:21.479446    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:21.597421 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:21.597396    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-88d98b7c6-w4njm"]
Jul 02 21:20:21.600338 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:20:21.600311    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca3427a_2e3a_496a_891c_326ea3e54437.slice/crio-6e89ddc71d694b52f83f63b8f8ed1d0987823e3a534822bbe78baddd6bf50c59 WatchSource:0}: Error finding container 6e89ddc71d694b52f83f63b8f8ed1d0987823e3a534822bbe78baddd6bf50c59: Status 404 returned error can't find the container with id 6e89ddc71d694b52f83f63b8f8ed1d0987823e3a534822bbe78baddd6bf50c59
Jul 02 21:20:22.333045 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.333009    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:20:22.348221 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.348192    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 21:20:22.352250 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.352229    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.352404 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.352380    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:22.355502 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.355476    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 02 21:20:22.355626 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.355477    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pvdwm\""
Jul 02 21:20:22.356070 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.356051    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9rfm01n9mndpd\""
Jul 02 21:20:22.356241 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.356059    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 02 21:20:22.356241 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.356088    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 02 21:20:22.356484 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.356088    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 02 21:20:22.356484 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.356090    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 02 21:20:22.356484 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.356129    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 02 21:20:22.356484 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.356145    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 02 21:20:22.356484 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.356424    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 02 21:20:22.356484 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.356216    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 02 21:20:22.356484 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.356214    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 02 21:20:22.356813 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.356587    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 02 21:20:22.356813 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.356720    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 02 21:20:22.356813 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.356775    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 02 21:20:22.359464 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.359445    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 02 21:20:22.374632 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.374611    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 21:20:22.444673 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.444648    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2" event={"ID":"f1ca83f6-4c9b-4317-8455-239386da3440","Type":"ContainerStarted","Data":"dfe93ced910c395ec4f0544c7d613ae9e792ac99339ee08a3488fc0fb9245999"}
Jul 02 21:20:22.444787 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.444713    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2" event={"ID":"f1ca83f6-4c9b-4317-8455-239386da3440","Type":"ContainerStarted","Data":"ebe9e809cfacd30491064a38128cb00984b39214e2f1d331b46a1968be5856b0"}
Jul 02 21:20:22.444787 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.444756    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2" event={"ID":"f1ca83f6-4c9b-4317-8455-239386da3440","Type":"ContainerStarted","Data":"62c505c152197cd5ac554d524b4b6d444c11cf2dd97a48449d0b9812f6ecb86a"}
Jul 02 21:20:22.446253 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.446221    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm" event={"ID":"0ca3427a-2e3a-496a-891c-326ea3e54437","Type":"ContainerStarted","Data":"6e89ddc71d694b52f83f63b8f8ed1d0987823e3a534822bbe78baddd6bf50c59"}
Jul 02 21:20:22.507260 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.507230    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.507581 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.507274    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.507581 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.507301    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.507581 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.507397    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.507581 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.507432    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.507581 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.507456    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvbwt\" (UniqueName: \"kubernetes.io/projected/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-kube-api-access-tvbwt\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.507581 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.507492    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.507581 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.507555    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-config-out\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.507916 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.507598    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.507916 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.507630    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.507916 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.507663    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.507916 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.507721    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-web-config\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.507916 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.507749    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.507916 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.507787    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-config\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.507916 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.507810    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.507916 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.507874    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.507916 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.507899    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.507916 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.507918    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.508333 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.507933    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.508333 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.507947    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.534505 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.534477    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:22.608355 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.608331    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.608469 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.608369    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-config\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.608469 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.608395    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.608469 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.608438    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.608591 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.608473    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.608591 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.608500    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.608591 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.608522    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.608591 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.608546    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.608591 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.608582    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.608760 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.608607    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.608760 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.608630    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.608760 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.608680    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.608760 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.608721    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.608760 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.608743    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvbwt\" (UniqueName: \"kubernetes.io/projected/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-kube-api-access-tvbwt\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.608926 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.608773    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.608926 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.608821    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-config-out\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.608926 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.608871    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.608926 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.608901    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.609051 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.608941    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.609051 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.608990    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-web-config\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.609122 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.609081    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.609957 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.609745    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.610063 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.610002    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.610129 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.610104    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.610961 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.610570    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.610961 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.610734    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.610961 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.610820    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.613415 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.613369    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-web-config\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.613516 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.613467    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.616120 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.615694    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.616120 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.615776    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.616120 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.615984    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.616120 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.615995    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.616120 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.616020    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.616120 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.616081    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.616416 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.616266    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-config-out\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.617520 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.617482    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.617610 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.617589    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.617976 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.617960    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-config\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.620455 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.620440    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvbwt\" (UniqueName: \"kubernetes.io/projected/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-kube-api-access-tvbwt\") pod \"prometheus-k8s-0\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.634632 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.634604    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:22.664067 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.664028    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:22.850121 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.850025    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:22.927289 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:22.927258    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:23.123475 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:23.123399    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 21:20:23.125439 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:20:23.125413    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce524c6_0854_484e_bbaf_a6cd593ab4cc.slice/crio-b05e8da790503d7916feee9b48b73b0a2e415caf7ba14e5f2e28c59c12ae94e6 WatchSource:0}: Error finding container b05e8da790503d7916feee9b48b73b0a2e415caf7ba14e5f2e28c59c12ae94e6: Status 404 returned error can't find the container with id b05e8da790503d7916feee9b48b73b0a2e415caf7ba14e5f2e28c59c12ae94e6
Jul 02 21:20:23.450860 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:23.450829    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ce524c6-0854-484e-bbaf-a6cd593ab4cc","Type":"ContainerStarted","Data":"b05e8da790503d7916feee9b48b73b0a2e415caf7ba14e5f2e28c59c12ae94e6"}
Jul 02 21:20:23.453803 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:23.453770    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2" event={"ID":"f1ca83f6-4c9b-4317-8455-239386da3440","Type":"ContainerStarted","Data":"253e4cc5c11c8ef5dc5a54feba9826f186e37416d1246be4a8dc52a6766a754f"}
Jul 02 21:20:23.453918 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:23.453832    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2" event={"ID":"f1ca83f6-4c9b-4317-8455-239386da3440","Type":"ContainerStarted","Data":"e7975a4823f00038f9e8423adf9bfbeb955a6572ad5f02963abbee5d5015a6c5"}
Jul 02 21:20:23.453918 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:23.453872    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2" event={"ID":"f1ca83f6-4c9b-4317-8455-239386da3440","Type":"ContainerStarted","Data":"2b2ae11168c574095062b217f52ec8c257e3f93287b98991d5b49d7b2ac728f2"}
Jul 02 21:20:23.454025 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:23.453920    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:23.455326 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:23.455306    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm" event={"ID":"0ca3427a-2e3a-496a-891c-326ea3e54437","Type":"ContainerStarted","Data":"6ce71ecc62b6805d4bdcea6df462cadaca5cd8c86ff390206eb7f3dbf1ef16ee"}
Jul 02 21:20:23.491813 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:23.491769    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2" podStartSLOduration=1.865754023 podStartE2EDuration="4.491755331s" totalImagesPullingTime="2.626001308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:23.490844014 +0000 UTC m=+95.117345892" watchObservedRunningTime="2026-07-02 21:20:23.491755331 +0000 UTC m=+95.118257237"
Jul 02 21:20:23.528025 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:23.527984    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm" podStartSLOduration=2.111541555 podStartE2EDuration="3.527971468s" totalImagesPullingTime="1.416429913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:23.527727144 +0000 UTC m=+95.154229039" watchObservedRunningTime="2026-07-02 21:20:23.527971468 +0000 UTC m=+95.154473344"
Jul 02 21:20:24.460439 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:24.460413    2575 generic.go:356] "Generic (PLEG): container finished" podID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerID="e43e5a64a62b450eb1579c9ef4cbd65e2408315fd69d4c81847f0d5fcdad2d92" exitCode=0
Jul 02 21:20:24.460533 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:24.460493    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ce524c6-0854-484e-bbaf-a6cd593ab4cc","Type":"ContainerDied","Data":"e43e5a64a62b450eb1579c9ef4cbd65e2408315fd69d4c81847f0d5fcdad2d92"}
Jul 02 21:20:25.478074 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:25.478034    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:25.579378 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:25.579341    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:25.646416 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:25.646371    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:27.673406 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:27.673385    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-647f4fc976-pzbv8"]
Jul 02 21:20:27.751466 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:27.751438    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:27.856472 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:27.856415    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:27.951396 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:27.951365    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:28.482260 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:28.482227    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ce524c6-0854-484e-bbaf-a6cd593ab4cc","Type":"ContainerStarted","Data":"a7d1823c94076139cf6ad1e81dec3d2b53dbae79fb134c0fda2c298789d646f4"}
Jul 02 21:20:28.482426 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:28.482315    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ce524c6-0854-484e-bbaf-a6cd593ab4cc","Type":"ContainerStarted","Data":"8fb54ee8d3997b0c5977931ef792a95eb6a0c2ff82de495fa746a6bf562157a7"}
Jul 02 21:20:28.482426 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:28.482344    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ce524c6-0854-484e-bbaf-a6cd593ab4cc","Type":"ContainerStarted","Data":"05eca4fab6e3007011b3f97c5c872f95dae772a4ed87b6b37ae6f38273b44fd9"}
Jul 02 21:20:28.482426 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:28.482367    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ce524c6-0854-484e-bbaf-a6cd593ab4cc","Type":"ContainerStarted","Data":"edafccd2fb9b261308b005f7d761a0549a55b32320d225f89231843c9fad20af"}
Jul 02 21:20:28.482426 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:28.482405    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ce524c6-0854-484e-bbaf-a6cd593ab4cc","Type":"ContainerStarted","Data":"b3d4586b9bed13b33cd7e8d221713ed59f2b710d2e3edc46d6251ef8b7d31706"}
Jul 02 21:20:28.482571 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:28.482431    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ce524c6-0854-484e-bbaf-a6cd593ab4cc","Type":"ContainerStarted","Data":"99e2e7936bcd89b39c13d781ea4fa19db9abbb00997e60f280ea3e78f5a1d4af"}
Jul 02 21:20:28.515286 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:28.515223    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.130947473 podStartE2EDuration="6.515202532s" totalImagesPullingTime="4.367074584s" totalInitContainerRuntime="17.180475ms" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:28.51308678 +0000 UTC m=+100.139588660" watchObservedRunningTime="2026-07-02 21:20:28.515202532 +0000 UTC m=+100.141704410"
Jul 02 21:20:29.410284 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:29.410248    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:29.467311 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:29.467283    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-bb85b44fd-l7bk2"
Jul 02 21:20:29.506431 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:29.506406    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:29.645276 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:29.645242    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:29.888861 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:29.888761    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:29.989585 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:29.989551    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:30.089962 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:30.089919    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:30.191676 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:30.191643    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:30.556579 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:30.556491    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:30.624142 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:30.624115    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:30.707373 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:30.707348    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:30.808144 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:30.808062    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:31.089097 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:31.089008    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:31.490479 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:31.490451    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:31.591540 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:31.591514    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:31.756463 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:31.756386    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:32.330442 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:32.330414    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b9tf7"
Jul 02 21:20:32.448330 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:32.448270    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:32.549090 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:32.549038    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:32.651146 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:32.651114    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:32.664782 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:32.664757    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:33.049557 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:33.049459    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:33.117082 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:33.117050    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:34.981609 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:34.981573    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:37.208427 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:37.208394    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:37.309238 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:37.309209    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:40.899600 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:40.899562    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:40.899962 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:40.899620    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:20:41.535361 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:41.535320    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:41.636384 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:41.636351    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:44.197414 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:44.197378    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:44.295263 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:44.295233    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:47.530484 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:47.530441    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:47.628282 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:47.628246    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:48.040621 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:48.040586    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:48.132350 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:48.132318    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:49.567641 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:49.567605    2575 generic.go:356] "Generic (PLEG): container finished" podID="78977169-1717-4b34-b8ac-7f4562f6be64" containerID="b2de0d017859b1c255d6ffaf122c2dfc57d9c044733f1e12efc4ff60ef0eedc1" exitCode=0
Jul 02 21:20:49.568105 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:49.567654    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-pbcv9" event={"ID":"78977169-1717-4b34-b8ac-7f4562f6be64","Type":"ContainerDied","Data":"b2de0d017859b1c255d6ffaf122c2dfc57d9c044733f1e12efc4ff60ef0eedc1"}
Jul 02 21:20:49.568105 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:49.568029    2575 scope.go:117] "RemoveContainer" containerID="b2de0d017859b1c255d6ffaf122c2dfc57d9c044733f1e12efc4ff60ef0eedc1"
Jul 02 21:20:50.573360 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:50.573326    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-pbcv9" event={"ID":"78977169-1717-4b34-b8ac-7f4562f6be64","Type":"ContainerStarted","Data":"6c8db7d769291e8aec29a047313deaf9f8b01cfc6f4bfc324ed54986f4780d21"}
Jul 02 21:20:52.693972 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:52.693912    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-647f4fc976-pzbv8" podUID="8d8e442e-419a-417c-bb3a-7227bf8e964f" containerName="registry" containerID="cri-o://d26ae13f5fe24eb16751d40424657e34b6843a66993687488be5f5647291ea02" gracePeriod=30
Jul 02 21:20:52.971759 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:52.971706    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:20:53.065092 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.065065    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-certificates\") pod \"8d8e442e-419a-417c-bb3a-7227bf8e964f\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") "
Jul 02 21:20:53.065253 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.065100    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d8e442e-419a-417c-bb3a-7227bf8e964f-trusted-ca\") pod \"8d8e442e-419a-417c-bb3a-7227bf8e964f\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") "
Jul 02 21:20:53.065253 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.065120    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-bound-sa-token\") pod \"8d8e442e-419a-417c-bb3a-7227bf8e964f\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") "
Jul 02 21:20:53.065253 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.065144    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d8e442e-419a-417c-bb3a-7227bf8e964f-image-registry-private-configuration\") pod \"8d8e442e-419a-417c-bb3a-7227bf8e964f\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") "
Jul 02 21:20:53.065253 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.065216    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vrss\" (UniqueName: \"kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-kube-api-access-6vrss\") pod \"8d8e442e-419a-417c-bb3a-7227bf8e964f\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") "
Jul 02 21:20:53.065461 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.065272    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls\") pod \"8d8e442e-419a-417c-bb3a-7227bf8e964f\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") "
Jul 02 21:20:53.065461 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.065306    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d8e442e-419a-417c-bb3a-7227bf8e964f-ca-trust-extracted\") pod \"8d8e442e-419a-417c-bb3a-7227bf8e964f\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") "
Jul 02 21:20:53.065461 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.065335    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d8e442e-419a-417c-bb3a-7227bf8e964f-installation-pull-secrets\") pod \"8d8e442e-419a-417c-bb3a-7227bf8e964f\" (UID: \"8d8e442e-419a-417c-bb3a-7227bf8e964f\") "
Jul 02 21:20:53.065617 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.065507    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d8e442e-419a-417c-bb3a-7227bf8e964f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8d8e442e-419a-417c-bb3a-7227bf8e964f" (UID: "8d8e442e-419a-417c-bb3a-7227bf8e964f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:20:53.065676 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.065617    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8d8e442e-419a-417c-bb3a-7227bf8e964f" (UID: "8d8e442e-419a-417c-bb3a-7227bf8e964f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:20:53.065676 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.065663    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d8e442e-419a-417c-bb3a-7227bf8e964f-trusted-ca\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:20:53.067863 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.067826    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8d8e442e-419a-417c-bb3a-7227bf8e964f" (UID: "8d8e442e-419a-417c-bb3a-7227bf8e964f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:20:53.068038 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.068012    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-kube-api-access-6vrss" (OuterVolumeSpecName: "kube-api-access-6vrss") pod "8d8e442e-419a-417c-bb3a-7227bf8e964f" (UID: "8d8e442e-419a-417c-bb3a-7227bf8e964f"). InnerVolumeSpecName "kube-api-access-6vrss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:20:53.068038 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.068019    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d8e442e-419a-417c-bb3a-7227bf8e964f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8d8e442e-419a-417c-bb3a-7227bf8e964f" (UID: "8d8e442e-419a-417c-bb3a-7227bf8e964f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:20:53.068152 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.068071    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8d8e442e-419a-417c-bb3a-7227bf8e964f" (UID: "8d8e442e-419a-417c-bb3a-7227bf8e964f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:20:53.068152 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.068114    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d8e442e-419a-417c-bb3a-7227bf8e964f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8d8e442e-419a-417c-bb3a-7227bf8e964f" (UID: "8d8e442e-419a-417c-bb3a-7227bf8e964f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:20:53.073321 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.073294    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8e442e-419a-417c-bb3a-7227bf8e964f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8d8e442e-419a-417c-bb3a-7227bf8e964f" (UID: "8d8e442e-419a-417c-bb3a-7227bf8e964f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:20:53.166451 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.166422    2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-certificates\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:20:53.166451 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.166449    2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-bound-sa-token\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:20:53.166599 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.166459    2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d8e442e-419a-417c-bb3a-7227bf8e964f-image-registry-private-configuration\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:20:53.166599 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.166468    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vrss\" (UniqueName: \"kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-kube-api-access-6vrss\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:20:53.166599 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.166478    2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d8e442e-419a-417c-bb3a-7227bf8e964f-registry-tls\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:20:53.166599 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.166486    2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d8e442e-419a-417c-bb3a-7227bf8e964f-ca-trust-extracted\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:20:53.166599 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.166493    2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d8e442e-419a-417c-bb3a-7227bf8e964f-installation-pull-secrets\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:20:53.587023 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.586987    2575 generic.go:356] "Generic (PLEG): container finished" podID="8d8e442e-419a-417c-bb3a-7227bf8e964f" containerID="d26ae13f5fe24eb16751d40424657e34b6843a66993687488be5f5647291ea02" exitCode=0
Jul 02 21:20:53.587216 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.587030    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-647f4fc976-pzbv8"
Jul 02 21:20:53.587216 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.587029    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-647f4fc976-pzbv8" event={"ID":"8d8e442e-419a-417c-bb3a-7227bf8e964f","Type":"ContainerDied","Data":"d26ae13f5fe24eb16751d40424657e34b6843a66993687488be5f5647291ea02"}
Jul 02 21:20:53.587216 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.587164    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-647f4fc976-pzbv8" event={"ID":"8d8e442e-419a-417c-bb3a-7227bf8e964f","Type":"ContainerDied","Data":"662dc792244fba8dce67f415b65065158615537072ff212ae344c952738b57cf"}
Jul 02 21:20:53.587416 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.587221    2575 scope.go:117] "RemoveContainer" containerID="d26ae13f5fe24eb16751d40424657e34b6843a66993687488be5f5647291ea02"
Jul 02 21:20:53.596289 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.596273    2575 scope.go:117] "RemoveContainer" containerID="d26ae13f5fe24eb16751d40424657e34b6843a66993687488be5f5647291ea02"
Jul 02 21:20:53.596568 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:20:53.596550    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d26ae13f5fe24eb16751d40424657e34b6843a66993687488be5f5647291ea02\": container with ID starting with d26ae13f5fe24eb16751d40424657e34b6843a66993687488be5f5647291ea02 not found: ID does not exist" containerID="d26ae13f5fe24eb16751d40424657e34b6843a66993687488be5f5647291ea02"
Jul 02 21:20:53.596617 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.596576    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d26ae13f5fe24eb16751d40424657e34b6843a66993687488be5f5647291ea02"} err="failed to get container status \"d26ae13f5fe24eb16751d40424657e34b6843a66993687488be5f5647291ea02\": rpc error: code = NotFound desc = could not find container \"d26ae13f5fe24eb16751d40424657e34b6843a66993687488be5f5647291ea02\": container with ID starting with d26ae13f5fe24eb16751d40424657e34b6843a66993687488be5f5647291ea02 not found: ID does not exist"
Jul 02 21:20:53.609092 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.609065    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-647f4fc976-pzbv8"]
Jul 02 21:20:53.613651 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:53.613630    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-647f4fc976-pzbv8"]
Jul 02 21:20:55.016466 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:55.016428    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d8e442e-419a-417c-bb3a-7227bf8e964f" path="/var/lib/kubelet/pods/8d8e442e-419a-417c-bb3a-7227bf8e964f/volumes"
Jul 02 21:20:55.114675 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:55.114648    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:55.214786 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:20:55.214758    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:00.904067 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:00.904038    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:21:00.907899 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:00.907878    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-88d98b7c6-w4njm"
Jul 02 21:21:03.035005 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:03.034971    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:03.102940 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:03.102909    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:07.127156 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:07.127123    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:07.224125 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:07.224096    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:11.103141 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:11.103111    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:22.664624 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:22.664590    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:22.686610 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:22.686578    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:22.714068 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:22.714044    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:35.360184 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:35.360140    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:36.288679 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:36.288650    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:40.737436 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:40.737349    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 21:21:40.737981 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:40.737737    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="prometheus" containerID="cri-o://99e2e7936bcd89b39c13d781ea4fa19db9abbb00997e60f280ea3e78f5a1d4af" gracePeriod=600
Jul 02 21:21:40.737981 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:40.737752    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a7d1823c94076139cf6ad1e81dec3d2b53dbae79fb134c0fda2c298789d646f4" gracePeriod=600
Jul 02 21:21:40.737981 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:40.737770    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="kube-rbac-proxy" containerID="cri-o://8fb54ee8d3997b0c5977931ef792a95eb6a0c2ff82de495fa746a6bf562157a7" gracePeriod=600
Jul 02 21:21:40.737981 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:40.737777    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="kube-rbac-proxy-web" containerID="cri-o://05eca4fab6e3007011b3f97c5c872f95dae772a4ed87b6b37ae6f38273b44fd9" gracePeriod=600
Jul 02 21:21:40.737981 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:40.737803    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="thanos-sidecar" containerID="cri-o://edafccd2fb9b261308b005f7d761a0549a55b32320d225f89231843c9fad20af" gracePeriod=600
Jul 02 21:21:40.737981 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:40.737819    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="config-reloader" containerID="cri-o://b3d4586b9bed13b33cd7e8d221713ed59f2b710d2e3edc46d6251ef8b7d31706" gracePeriod=600
Jul 02 21:21:41.774605 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:41.774570    2575 generic.go:356] "Generic (PLEG): container finished" podID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerID="a7d1823c94076139cf6ad1e81dec3d2b53dbae79fb134c0fda2c298789d646f4" exitCode=0
Jul 02 21:21:41.774605 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:41.774595    2575 generic.go:356] "Generic (PLEG): container finished" podID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerID="8fb54ee8d3997b0c5977931ef792a95eb6a0c2ff82de495fa746a6bf562157a7" exitCode=0
Jul 02 21:21:41.774605 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:41.774602    2575 generic.go:356] "Generic (PLEG): container finished" podID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerID="edafccd2fb9b261308b005f7d761a0549a55b32320d225f89231843c9fad20af" exitCode=0
Jul 02 21:21:41.774605 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:41.774608    2575 generic.go:356] "Generic (PLEG): container finished" podID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerID="b3d4586b9bed13b33cd7e8d221713ed59f2b710d2e3edc46d6251ef8b7d31706" exitCode=0
Jul 02 21:21:41.774605 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:41.774613    2575 generic.go:356] "Generic (PLEG): container finished" podID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerID="99e2e7936bcd89b39c13d781ea4fa19db9abbb00997e60f280ea3e78f5a1d4af" exitCode=0
Jul 02 21:21:41.775112 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:41.774639    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ce524c6-0854-484e-bbaf-a6cd593ab4cc","Type":"ContainerDied","Data":"a7d1823c94076139cf6ad1e81dec3d2b53dbae79fb134c0fda2c298789d646f4"}
Jul 02 21:21:41.775112 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:41.774701    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ce524c6-0854-484e-bbaf-a6cd593ab4cc","Type":"ContainerDied","Data":"8fb54ee8d3997b0c5977931ef792a95eb6a0c2ff82de495fa746a6bf562157a7"}
Jul 02 21:21:41.775112 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:41.774737    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ce524c6-0854-484e-bbaf-a6cd593ab4cc","Type":"ContainerDied","Data":"edafccd2fb9b261308b005f7d761a0549a55b32320d225f89231843c9fad20af"}
Jul 02 21:21:41.775112 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:41.774763    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ce524c6-0854-484e-bbaf-a6cd593ab4cc","Type":"ContainerDied","Data":"b3d4586b9bed13b33cd7e8d221713ed59f2b710d2e3edc46d6251ef8b7d31706"}
Jul 02 21:21:41.775112 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:41.774786    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ce524c6-0854-484e-bbaf-a6cd593ab4cc","Type":"ContainerDied","Data":"99e2e7936bcd89b39c13d781ea4fa19db9abbb00997e60f280ea3e78f5a1d4af"}
Jul 02 21:21:41.988470 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:41.988450    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.048046 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.047982    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-kube-rbac-proxy\") pod \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") "
Jul 02 21:21:42.048046 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.048023    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-grpc-tls\") pod \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") "
Jul 02 21:21:42.048245 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.048050    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-config-out\") pod \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") "
Jul 02 21:21:42.048245 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.048074    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-config\") pod \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") "
Jul 02 21:21:42.048245 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.048113    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-configmap-serving-certs-ca-bundle\") pod \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") "
Jul 02 21:21:42.048245 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.048143    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-configmap-metrics-client-ca\") pod \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") "
Jul 02 21:21:42.048245 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.048184    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-rulefiles-1\") pod \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") "
Jul 02 21:21:42.048245 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.048209    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-db\") pod \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") "
Jul 02 21:21:42.048245 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.048240    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") "
Jul 02 21:21:42.048584 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.048272    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-web-config\") pod \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") "
Jul 02 21:21:42.048584 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.048309    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-rulefiles-0\") pod \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") "
Jul 02 21:21:42.048584 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.048346    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-tls-assets\") pod \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") "
Jul 02 21:21:42.048584 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.048382    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvbwt\" (UniqueName: \"kubernetes.io/projected/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-kube-api-access-tvbwt\") pod \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") "
Jul 02 21:21:42.048584 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.048407    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") "
Jul 02 21:21:42.048584 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.048444    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-trusted-ca-bundle\") pod \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") "
Jul 02 21:21:42.048584 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.048469    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-metrics-client-certs\") pod \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") "
Jul 02 21:21:42.048584 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.048501    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-rulefiles-2\") pod \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") "
Jul 02 21:21:42.048584 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.048542    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-configmap-kubelet-serving-ca-bundle\") pod \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") "
Jul 02 21:21:42.048584 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.048571    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-thanos-prometheus-http-client-file\") pod \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") "
Jul 02 21:21:42.049075 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.048605    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-prometheus-k8s-tls\") pod \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\" (UID: \"4ce524c6-0854-484e-bbaf-a6cd593ab4cc\") "
Jul 02 21:21:42.049075 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.048709    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "4ce524c6-0854-484e-bbaf-a6cd593ab4cc" (UID: "4ce524c6-0854-484e-bbaf-a6cd593ab4cc"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:21:42.049075 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.048851    2575 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-configmap-metrics-client-ca\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:21:42.049291 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.049253    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "4ce524c6-0854-484e-bbaf-a6cd593ab4cc" (UID: "4ce524c6-0854-484e-bbaf-a6cd593ab4cc"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:21:42.049409 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.049261    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "4ce524c6-0854-484e-bbaf-a6cd593ab4cc" (UID: "4ce524c6-0854-484e-bbaf-a6cd593ab4cc"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:21:42.049740 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.049715    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "4ce524c6-0854-484e-bbaf-a6cd593ab4cc" (UID: "4ce524c6-0854-484e-bbaf-a6cd593ab4cc"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:21:42.049864 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.049727    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "4ce524c6-0854-484e-bbaf-a6cd593ab4cc" (UID: "4ce524c6-0854-484e-bbaf-a6cd593ab4cc"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:21:42.050051 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.050028    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "4ce524c6-0854-484e-bbaf-a6cd593ab4cc" (UID: "4ce524c6-0854-484e-bbaf-a6cd593ab4cc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:21:42.050480 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.050458    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "4ce524c6-0854-484e-bbaf-a6cd593ab4cc" (UID: "4ce524c6-0854-484e-bbaf-a6cd593ab4cc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:21:42.050926 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.050854    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-config-out" (OuterVolumeSpecName: "config-out") pod "4ce524c6-0854-484e-bbaf-a6cd593ab4cc" (UID: "4ce524c6-0854-484e-bbaf-a6cd593ab4cc"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:21:42.051193 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.050974    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-config" (OuterVolumeSpecName: "config") pod "4ce524c6-0854-484e-bbaf-a6cd593ab4cc" (UID: "4ce524c6-0854-484e-bbaf-a6cd593ab4cc"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:42.051369 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.051347    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "4ce524c6-0854-484e-bbaf-a6cd593ab4cc" (UID: "4ce524c6-0854-484e-bbaf-a6cd593ab4cc"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:42.052207 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.051913    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "4ce524c6-0854-484e-bbaf-a6cd593ab4cc" (UID: "4ce524c6-0854-484e-bbaf-a6cd593ab4cc"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:42.052207 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.052182    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "4ce524c6-0854-484e-bbaf-a6cd593ab4cc" (UID: "4ce524c6-0854-484e-bbaf-a6cd593ab4cc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:21:42.052770 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.052706    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "4ce524c6-0854-484e-bbaf-a6cd593ab4cc" (UID: "4ce524c6-0854-484e-bbaf-a6cd593ab4cc"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:42.053163 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.053102    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "4ce524c6-0854-484e-bbaf-a6cd593ab4cc" (UID: "4ce524c6-0854-484e-bbaf-a6cd593ab4cc"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:42.053270 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.053215    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4ce524c6-0854-484e-bbaf-a6cd593ab4cc" (UID: "4ce524c6-0854-484e-bbaf-a6cd593ab4cc"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:21:42.053270 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.053242    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "4ce524c6-0854-484e-bbaf-a6cd593ab4cc" (UID: "4ce524c6-0854-484e-bbaf-a6cd593ab4cc"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:42.053678 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.053661    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "4ce524c6-0854-484e-bbaf-a6cd593ab4cc" (UID: "4ce524c6-0854-484e-bbaf-a6cd593ab4cc"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:42.054245 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.054217    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-kube-api-access-tvbwt" (OuterVolumeSpecName: "kube-api-access-tvbwt") pod "4ce524c6-0854-484e-bbaf-a6cd593ab4cc" (UID: "4ce524c6-0854-484e-bbaf-a6cd593ab4cc"). InnerVolumeSpecName "kube-api-access-tvbwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:21:42.054834 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.054813    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "4ce524c6-0854-484e-bbaf-a6cd593ab4cc" (UID: "4ce524c6-0854-484e-bbaf-a6cd593ab4cc"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:42.066025 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.066000    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-web-config" (OuterVolumeSpecName: "web-config") pod "4ce524c6-0854-484e-bbaf-a6cd593ab4cc" (UID: "4ce524c6-0854-484e-bbaf-a6cd593ab4cc"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:42.149150 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.149116    2575 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-tls-assets\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:21:42.149150 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.149144    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvbwt\" (UniqueName: \"kubernetes.io/projected/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-kube-api-access-tvbwt\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:21:42.149150 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.149157    2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:21:42.149380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.149185    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:21:42.149380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.149195    2575 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-metrics-client-certs\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:21:42.149380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.149204    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:21:42.149380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.149213    2575 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:21:42.149380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.149221    2575 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:21:42.149380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.149230    2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:21:42.149380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.149237    2575 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-kube-rbac-proxy\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:21:42.149380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.149246    2575 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-grpc-tls\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:21:42.149380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.149254    2575 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-config-out\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:21:42.149380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.149262    2575 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-config\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:21:42.149380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.149270    2575 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:21:42.149380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.149278    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:21:42.149380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.149288    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-db\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:21:42.149380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.149295    2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:21:42.149380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.149303    2575 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-web-config\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:21:42.149380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.149311    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4ce524c6-0854-484e-bbaf-a6cd593ab4cc-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:21:42.781331 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.781296    2575 generic.go:356] "Generic (PLEG): container finished" podID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerID="05eca4fab6e3007011b3f97c5c872f95dae772a4ed87b6b37ae6f38273b44fd9" exitCode=0
Jul 02 21:21:42.781711 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.781376    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.781711 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.781373    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ce524c6-0854-484e-bbaf-a6cd593ab4cc","Type":"ContainerDied","Data":"05eca4fab6e3007011b3f97c5c872f95dae772a4ed87b6b37ae6f38273b44fd9"}
Jul 02 21:21:42.781711 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.781469    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ce524c6-0854-484e-bbaf-a6cd593ab4cc","Type":"ContainerDied","Data":"b05e8da790503d7916feee9b48b73b0a2e415caf7ba14e5f2e28c59c12ae94e6"}
Jul 02 21:21:42.781711 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.781506    2575 scope.go:117] "RemoveContainer" containerID="a7d1823c94076139cf6ad1e81dec3d2b53dbae79fb134c0fda2c298789d646f4"
Jul 02 21:21:42.789207 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.789191    2575 scope.go:117] "RemoveContainer" containerID="8fb54ee8d3997b0c5977931ef792a95eb6a0c2ff82de495fa746a6bf562157a7"
Jul 02 21:21:42.797005 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.796988    2575 scope.go:117] "RemoveContainer" containerID="05eca4fab6e3007011b3f97c5c872f95dae772a4ed87b6b37ae6f38273b44fd9"
Jul 02 21:21:42.803580 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.803562    2575 scope.go:117] "RemoveContainer" containerID="edafccd2fb9b261308b005f7d761a0549a55b32320d225f89231843c9fad20af"
Jul 02 21:21:42.806569 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.806546    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 21:21:42.810896 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.810880    2575 scope.go:117] "RemoveContainer" containerID="b3d4586b9bed13b33cd7e8d221713ed59f2b710d2e3edc46d6251ef8b7d31706"
Jul 02 21:21:42.811739 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.811686    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 21:21:42.817462 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.817447    2575 scope.go:117] "RemoveContainer" containerID="99e2e7936bcd89b39c13d781ea4fa19db9abbb00997e60f280ea3e78f5a1d4af"
Jul 02 21:21:42.827562 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.827545    2575 scope.go:117] "RemoveContainer" containerID="e43e5a64a62b450eb1579c9ef4cbd65e2408315fd69d4c81847f0d5fcdad2d92"
Jul 02 21:21:42.833982 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.833967    2575 scope.go:117] "RemoveContainer" containerID="a7d1823c94076139cf6ad1e81dec3d2b53dbae79fb134c0fda2c298789d646f4"
Jul 02 21:21:42.834255 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:21:42.834238    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d1823c94076139cf6ad1e81dec3d2b53dbae79fb134c0fda2c298789d646f4\": container with ID starting with a7d1823c94076139cf6ad1e81dec3d2b53dbae79fb134c0fda2c298789d646f4 not found: ID does not exist" containerID="a7d1823c94076139cf6ad1e81dec3d2b53dbae79fb134c0fda2c298789d646f4"
Jul 02 21:21:42.834324 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.834265    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d1823c94076139cf6ad1e81dec3d2b53dbae79fb134c0fda2c298789d646f4"} err="failed to get container status \"a7d1823c94076139cf6ad1e81dec3d2b53dbae79fb134c0fda2c298789d646f4\": rpc error: code = NotFound desc = could not find container \"a7d1823c94076139cf6ad1e81dec3d2b53dbae79fb134c0fda2c298789d646f4\": container with ID starting with a7d1823c94076139cf6ad1e81dec3d2b53dbae79fb134c0fda2c298789d646f4 not found: ID does not exist"
Jul 02 21:21:42.834324 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.834283    2575 scope.go:117] "RemoveContainer" containerID="8fb54ee8d3997b0c5977931ef792a95eb6a0c2ff82de495fa746a6bf562157a7"
Jul 02 21:21:42.834564 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:21:42.834542    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb54ee8d3997b0c5977931ef792a95eb6a0c2ff82de495fa746a6bf562157a7\": container with ID starting with 8fb54ee8d3997b0c5977931ef792a95eb6a0c2ff82de495fa746a6bf562157a7 not found: ID does not exist" containerID="8fb54ee8d3997b0c5977931ef792a95eb6a0c2ff82de495fa746a6bf562157a7"
Jul 02 21:21:42.834655 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.834568    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb54ee8d3997b0c5977931ef792a95eb6a0c2ff82de495fa746a6bf562157a7"} err="failed to get container status \"8fb54ee8d3997b0c5977931ef792a95eb6a0c2ff82de495fa746a6bf562157a7\": rpc error: code = NotFound desc = could not find container \"8fb54ee8d3997b0c5977931ef792a95eb6a0c2ff82de495fa746a6bf562157a7\": container with ID starting with 8fb54ee8d3997b0c5977931ef792a95eb6a0c2ff82de495fa746a6bf562157a7 not found: ID does not exist"
Jul 02 21:21:42.834655 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.834585    2575 scope.go:117] "RemoveContainer" containerID="05eca4fab6e3007011b3f97c5c872f95dae772a4ed87b6b37ae6f38273b44fd9"
Jul 02 21:21:42.834808 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:21:42.834792    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05eca4fab6e3007011b3f97c5c872f95dae772a4ed87b6b37ae6f38273b44fd9\": container with ID starting with 05eca4fab6e3007011b3f97c5c872f95dae772a4ed87b6b37ae6f38273b44fd9 not found: ID does not exist" containerID="05eca4fab6e3007011b3f97c5c872f95dae772a4ed87b6b37ae6f38273b44fd9"
Jul 02 21:21:42.834847 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.834813    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05eca4fab6e3007011b3f97c5c872f95dae772a4ed87b6b37ae6f38273b44fd9"} err="failed to get container status \"05eca4fab6e3007011b3f97c5c872f95dae772a4ed87b6b37ae6f38273b44fd9\": rpc error: code = NotFound desc = could not find container \"05eca4fab6e3007011b3f97c5c872f95dae772a4ed87b6b37ae6f38273b44fd9\": container with ID starting with 05eca4fab6e3007011b3f97c5c872f95dae772a4ed87b6b37ae6f38273b44fd9 not found: ID does not exist"
Jul 02 21:21:42.834847 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.834827    2575 scope.go:117] "RemoveContainer" containerID="edafccd2fb9b261308b005f7d761a0549a55b32320d225f89231843c9fad20af"
Jul 02 21:21:42.835079 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:21:42.835062    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edafccd2fb9b261308b005f7d761a0549a55b32320d225f89231843c9fad20af\": container with ID starting with edafccd2fb9b261308b005f7d761a0549a55b32320d225f89231843c9fad20af not found: ID does not exist" containerID="edafccd2fb9b261308b005f7d761a0549a55b32320d225f89231843c9fad20af"
Jul 02 21:21:42.835124 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.835082    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edafccd2fb9b261308b005f7d761a0549a55b32320d225f89231843c9fad20af"} err="failed to get container status \"edafccd2fb9b261308b005f7d761a0549a55b32320d225f89231843c9fad20af\": rpc error: code = NotFound desc = could not find container \"edafccd2fb9b261308b005f7d761a0549a55b32320d225f89231843c9fad20af\": container with ID starting with edafccd2fb9b261308b005f7d761a0549a55b32320d225f89231843c9fad20af not found: ID does not exist"
Jul 02 21:21:42.835124 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.835095    2575 scope.go:117] "RemoveContainer" containerID="b3d4586b9bed13b33cd7e8d221713ed59f2b710d2e3edc46d6251ef8b7d31706"
Jul 02 21:21:42.835300 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:21:42.835287    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3d4586b9bed13b33cd7e8d221713ed59f2b710d2e3edc46d6251ef8b7d31706\": container with ID starting with b3d4586b9bed13b33cd7e8d221713ed59f2b710d2e3edc46d6251ef8b7d31706 not found: ID does not exist" containerID="b3d4586b9bed13b33cd7e8d221713ed59f2b710d2e3edc46d6251ef8b7d31706"
Jul 02 21:21:42.835346 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.835302    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3d4586b9bed13b33cd7e8d221713ed59f2b710d2e3edc46d6251ef8b7d31706"} err="failed to get container status \"b3d4586b9bed13b33cd7e8d221713ed59f2b710d2e3edc46d6251ef8b7d31706\": rpc error: code = NotFound desc = could not find container \"b3d4586b9bed13b33cd7e8d221713ed59f2b710d2e3edc46d6251ef8b7d31706\": container with ID starting with b3d4586b9bed13b33cd7e8d221713ed59f2b710d2e3edc46d6251ef8b7d31706 not found: ID does not exist"
Jul 02 21:21:42.835346 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.835314    2575 scope.go:117] "RemoveContainer" containerID="99e2e7936bcd89b39c13d781ea4fa19db9abbb00997e60f280ea3e78f5a1d4af"
Jul 02 21:21:42.835542 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:21:42.835528    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99e2e7936bcd89b39c13d781ea4fa19db9abbb00997e60f280ea3e78f5a1d4af\": container with ID starting with 99e2e7936bcd89b39c13d781ea4fa19db9abbb00997e60f280ea3e78f5a1d4af not found: ID does not exist" containerID="99e2e7936bcd89b39c13d781ea4fa19db9abbb00997e60f280ea3e78f5a1d4af"
Jul 02 21:21:42.835584 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.835545    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99e2e7936bcd89b39c13d781ea4fa19db9abbb00997e60f280ea3e78f5a1d4af"} err="failed to get container status \"99e2e7936bcd89b39c13d781ea4fa19db9abbb00997e60f280ea3e78f5a1d4af\": rpc error: code = NotFound desc = could not find container \"99e2e7936bcd89b39c13d781ea4fa19db9abbb00997e60f280ea3e78f5a1d4af\": container with ID starting with 99e2e7936bcd89b39c13d781ea4fa19db9abbb00997e60f280ea3e78f5a1d4af not found: ID does not exist"
Jul 02 21:21:42.835584 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.835558    2575 scope.go:117] "RemoveContainer" containerID="e43e5a64a62b450eb1579c9ef4cbd65e2408315fd69d4c81847f0d5fcdad2d92"
Jul 02 21:21:42.835765 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:21:42.835748    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e43e5a64a62b450eb1579c9ef4cbd65e2408315fd69d4c81847f0d5fcdad2d92\": container with ID starting with e43e5a64a62b450eb1579c9ef4cbd65e2408315fd69d4c81847f0d5fcdad2d92 not found: ID does not exist" containerID="e43e5a64a62b450eb1579c9ef4cbd65e2408315fd69d4c81847f0d5fcdad2d92"
Jul 02 21:21:42.835807 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.835770    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e43e5a64a62b450eb1579c9ef4cbd65e2408315fd69d4c81847f0d5fcdad2d92"} err="failed to get container status \"e43e5a64a62b450eb1579c9ef4cbd65e2408315fd69d4c81847f0d5fcdad2d92\": rpc error: code = NotFound desc = could not find container \"e43e5a64a62b450eb1579c9ef4cbd65e2408315fd69d4c81847f0d5fcdad2d92\": container with ID starting with e43e5a64a62b450eb1579c9ef4cbd65e2408315fd69d4c81847f0d5fcdad2d92 not found: ID does not exist"
Jul 02 21:21:42.838728 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.838709    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 21:21:42.838983 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.838971    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d8e442e-419a-417c-bb3a-7227bf8e964f" containerName="registry"
Jul 02 21:21:42.839023 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.838985    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d8e442e-419a-417c-bb3a-7227bf8e964f" containerName="registry"
Jul 02 21:21:42.839023 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.838993    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="thanos-sidecar"
Jul 02 21:21:42.839023 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.838999    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="thanos-sidecar"
Jul 02 21:21:42.839023 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.839011    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="prometheus"
Jul 02 21:21:42.839023 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.839017    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="prometheus"
Jul 02 21:21:42.839294 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.839025    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="kube-rbac-proxy-web"
Jul 02 21:21:42.839294 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.839030    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="kube-rbac-proxy-web"
Jul 02 21:21:42.839294 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.839038    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="init-config-reloader"
Jul 02 21:21:42.839294 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.839043    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="init-config-reloader"
Jul 02 21:21:42.839294 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.839050    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="kube-rbac-proxy-thanos"
Jul 02 21:21:42.839294 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.839055    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="kube-rbac-proxy-thanos"
Jul 02 21:21:42.839294 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.839065    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="config-reloader"
Jul 02 21:21:42.839294 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.839070    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="config-reloader"
Jul 02 21:21:42.839294 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.839077    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="kube-rbac-proxy"
Jul 02 21:21:42.839294 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.839081    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="kube-rbac-proxy"
Jul 02 21:21:42.839294 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.839123    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="kube-rbac-proxy-web"
Jul 02 21:21:42.839294 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.839133    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="kube-rbac-proxy-thanos"
Jul 02 21:21:42.839294 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.839139    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="kube-rbac-proxy"
Jul 02 21:21:42.839294 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.839144    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d8e442e-419a-417c-bb3a-7227bf8e964f" containerName="registry"
Jul 02 21:21:42.839294 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.839150    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="prometheus"
Jul 02 21:21:42.839294 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.839155    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="config-reloader"
Jul 02 21:21:42.839294 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.839161    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" containerName="thanos-sidecar"
Jul 02 21:21:42.844407 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.844393    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.846972 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.846954    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 02 21:21:42.847308 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.847293    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 02 21:21:42.847496 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.847478    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 02 21:21:42.847590 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.847482    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 02 21:21:42.847590 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.847547    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 02 21:21:42.847707 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.847598    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 02 21:21:42.847707 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.847559    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 02 21:21:42.847808 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.847709    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 02 21:21:42.847993 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.847977    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 02 21:21:42.848067 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.848041    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9rfm01n9mndpd\""
Jul 02 21:21:42.848341 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.848322    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 02 21:21:42.848436 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.848342    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 02 21:21:42.848436 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.848331    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 02 21:21:42.848637 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.848622    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pvdwm\""
Jul 02 21:21:42.850861 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.850837    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 02 21:21:42.852370 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.852349    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 02 21:21:42.854569 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.854551    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.854649 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.854576    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4773d17b-6ca7-4009-902f-7575c1d4b201-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.854649 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.854593    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-config\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.854649 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.854608    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-web-config\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.854762 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.854665    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4773d17b-6ca7-4009-902f-7575c1d4b201-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.854762 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.854689    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4773d17b-6ca7-4009-902f-7575c1d4b201-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.854762 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.854713    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4773d17b-6ca7-4009-902f-7575c1d4b201-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.854762 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.854739    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4773d17b-6ca7-4009-902f-7575c1d4b201-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.854923 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.854775    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.854923 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.854804    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.854923 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.854818    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.854923 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.854849    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.854923 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.854883    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hsp5\" (UniqueName: \"kubernetes.io/projected/4773d17b-6ca7-4009-902f-7575c1d4b201-kube-api-access-7hsp5\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.855137 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.854927    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4773d17b-6ca7-4009-902f-7575c1d4b201-config-out\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.855137 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.854960    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.855137 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.854990    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4773d17b-6ca7-4009-902f-7575c1d4b201-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.855137 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.855027    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4773d17b-6ca7-4009-902f-7575c1d4b201-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.855137 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.855053    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.855137 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.855075    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4773d17b-6ca7-4009-902f-7575c1d4b201-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.855137 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.855096    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4773d17b-6ca7-4009-902f-7575c1d4b201-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.862934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.862912    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 21:21:42.956034 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.956009    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.956128 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.956040    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.956128 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.956061    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.956128 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.956074    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hsp5\" (UniqueName: \"kubernetes.io/projected/4773d17b-6ca7-4009-902f-7575c1d4b201-kube-api-access-7hsp5\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.956349 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.956328    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4773d17b-6ca7-4009-902f-7575c1d4b201-config-out\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.956414 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.956375    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.956414 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.956404    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4773d17b-6ca7-4009-902f-7575c1d4b201-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.956526 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.956421    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4773d17b-6ca7-4009-902f-7575c1d4b201-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.956526 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.956449    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.956526 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.956476    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4773d17b-6ca7-4009-902f-7575c1d4b201-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.956526 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.956505    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4773d17b-6ca7-4009-902f-7575c1d4b201-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.956717 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.956542    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.956717 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.956568    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4773d17b-6ca7-4009-902f-7575c1d4b201-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.956717 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.956594    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-config\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.956717 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.956617    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-web-config\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.956717 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.956653    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4773d17b-6ca7-4009-902f-7575c1d4b201-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.956717 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.956681    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4773d17b-6ca7-4009-902f-7575c1d4b201-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.956717 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.956715    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4773d17b-6ca7-4009-902f-7575c1d4b201-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.957060 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.956762    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4773d17b-6ca7-4009-902f-7575c1d4b201-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.957060 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.956809    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.958955 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.957388    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4773d17b-6ca7-4009-902f-7575c1d4b201-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.958955 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.957623    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4773d17b-6ca7-4009-902f-7575c1d4b201-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.958955 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.958265    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4773d17b-6ca7-4009-902f-7575c1d4b201-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.958955 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.958923    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4773d17b-6ca7-4009-902f-7575c1d4b201-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.959160 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.959059    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.959277 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.959256    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4773d17b-6ca7-4009-902f-7575c1d4b201-config-out\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.960019 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.959440    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.960019 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.959542    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4773d17b-6ca7-4009-902f-7575c1d4b201-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.960019 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.959878    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4773d17b-6ca7-4009-902f-7575c1d4b201-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.960019 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.959980    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4773d17b-6ca7-4009-902f-7575c1d4b201-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.960741 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.960701    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.961297 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.961236    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.961297 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.961251    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-config\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.961297 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.961246    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.961491 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.961450    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4773d17b-6ca7-4009-902f-7575c1d4b201-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.961491 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.961486    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-web-config\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.961640 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.961621    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.962103 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.962088    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4773d17b-6ca7-4009-902f-7575c1d4b201-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.962623 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.962608    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4773d17b-6ca7-4009-902f-7575c1d4b201-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:42.964607 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:42.964591    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hsp5\" (UniqueName: \"kubernetes.io/projected/4773d17b-6ca7-4009-902f-7575c1d4b201-kube-api-access-7hsp5\") pod \"prometheus-k8s-0\" (UID: \"4773d17b-6ca7-4009-902f-7575c1d4b201\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:43.013687 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:43.013667    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ce524c6-0854-484e-bbaf-a6cd593ab4cc" path="/var/lib/kubelet/pods/4ce524c6-0854-484e-bbaf-a6cd593ab4cc/volumes"
Jul 02 21:21:43.154618 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:43.154598    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:43.289137 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:43.289114    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 21:21:43.291473 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:21:43.291450    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4773d17b_6ca7_4009_902f_7575c1d4b201.slice/crio-e56edd6c02ba82cadb75b9de24ef5d6ce44229044a8d2e752c16382b807c7ea2 WatchSource:0}: Error finding container e56edd6c02ba82cadb75b9de24ef5d6ce44229044a8d2e752c16382b807c7ea2: Status 404 returned error can't find the container with id e56edd6c02ba82cadb75b9de24ef5d6ce44229044a8d2e752c16382b807c7ea2
Jul 02 21:21:43.788447 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:43.788411    2575 generic.go:356] "Generic (PLEG): container finished" podID="4773d17b-6ca7-4009-902f-7575c1d4b201" containerID="296f454cdfc0d1f436a2bcf9937e87fab910ec73f9be4ce42fc4ad12c6c63551" exitCode=0
Jul 02 21:21:43.788843 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:43.788505    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4773d17b-6ca7-4009-902f-7575c1d4b201","Type":"ContainerDied","Data":"296f454cdfc0d1f436a2bcf9937e87fab910ec73f9be4ce42fc4ad12c6c63551"}
Jul 02 21:21:43.788843 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:43.788608    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4773d17b-6ca7-4009-902f-7575c1d4b201","Type":"ContainerStarted","Data":"e56edd6c02ba82cadb75b9de24ef5d6ce44229044a8d2e752c16382b807c7ea2"}
Jul 02 21:21:43.917681 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:43.917648    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:44.017446 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:44.017381    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:44.796275 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:44.796232    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4773d17b-6ca7-4009-902f-7575c1d4b201","Type":"ContainerStarted","Data":"df9df833652355b3aa02b78b8932997917418620bbf81383f680fc0c91f9da62"}
Jul 02 21:21:44.796687 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:44.796321    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4773d17b-6ca7-4009-902f-7575c1d4b201","Type":"ContainerStarted","Data":"11816d9f6ec945d238a2bf706eb17ef5bcc528947a42cecb85e2fdc1ba053d2f"}
Jul 02 21:21:44.796687 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:44.796362    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4773d17b-6ca7-4009-902f-7575c1d4b201","Type":"ContainerStarted","Data":"bae56791b03a4fa3fba9b7d840b11a8225a2319854f23507c3ffadf228ff448c"}
Jul 02 21:21:44.796687 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:44.796399    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4773d17b-6ca7-4009-902f-7575c1d4b201","Type":"ContainerStarted","Data":"27bbaa655092d9278ea7ccced325f25b7975fd31d2e291abcc5d845291ea0910"}
Jul 02 21:21:44.796687 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:44.796442    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4773d17b-6ca7-4009-902f-7575c1d4b201","Type":"ContainerStarted","Data":"214a749b69b03e7686bdfc3686f44704a5ad7e067d5c46bb68aafc79000e6360"}
Jul 02 21:21:44.796687 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:44.796476    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4773d17b-6ca7-4009-902f-7575c1d4b201","Type":"ContainerStarted","Data":"4e81a9991a9d2ee6c0038434291d28fed9a25c423f58a07507a4a3ae107348ab"}
Jul 02 21:21:44.828291 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:44.828244    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.809163393 podStartE2EDuration="2.82822879s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.065397ms" isStatefulPod=false podCreationTimestamp="2026-07-02 21:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:21:44.826070156 +0000 UTC m=+176.452572056" watchObservedRunningTime="2026-07-02 21:21:44.82822879 +0000 UTC m=+176.454730667"
Jul 02 21:21:48.155558 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:48.155521    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:55.286288 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:55.286254    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:55.386612 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:55.386585    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:55.444990 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:21:55.444960    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:22:01.663220 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:22:01.663185    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:22:38.577114 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:22:38.577073    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:22:38.638494 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:22:38.638460    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:22:43.154799 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:22:43.154764    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:22:43.169779 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:22:43.169751    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:22:44.036244 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:22:44.036220    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:22:50.594132 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:22:50.594096    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:22:50.659523 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:22:50.659489    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:22:51.174816 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:22:51.174788    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:22:51.234593 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:22:51.234566    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:23:02.518761 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:23:02.518726    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:23:02.585918 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:23:02.585884    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:23:12.129439 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:23:12.129365    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:23:12.230290 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:23:12.230265    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:23:24.186904 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:23:24.186865    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:23:24.245799 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:23:24.245775    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:23:34.634067 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:23:34.634029    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:23:34.733520 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:23:34.733495    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:23:34.834137 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:23:34.834116    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:23:48.773512 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:23:48.773489    2575 kubelet.go:1648] "Image garbage collection succeeded"
Jul 02 21:24:11.782299 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:24:11.782266    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:24:12.996139 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:24:12.996100    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:24:44.979323 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:24:44.979243    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:24:45.087824 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:24:45.087782    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:24:51.127442 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:24:51.127409    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:24:51.396971 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:24:51.396935    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:24:51.599951 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:24:51.599918    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:24:51.696154 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:24:51.696064    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:24:51.797350 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:24:51.797316    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:24:52.099652 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:24:52.099576    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:24:52.180834 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:24:52.180804    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:24:52.484575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:24:52.484540    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:24:52.594360 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:24:52.594321    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:24:52.686674 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:24:52.686624    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:24:52.753476 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:24:52.753375    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:00.017280 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:26:00.017239    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:00.101643 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:26:00.101607    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:00.201401 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:26:00.201367    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:00.302311 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:26:00.302222    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:00.582227 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:26:00.582139    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:00.682908 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:26:00.682875    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:00.745004 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:26:00.744976    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:25.679084 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:26:25.679048    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:42.277100 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:26:42.277062    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:42.335283 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:26:42.335248    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:45.612887 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:26:45.612854    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:45.678842 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:26:45.678815    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:56.463771 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:26:56.463729    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:56.565001 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:26:56.564974    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:56.623436 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:26:56.623403    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:27:09.722239 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:09.722201    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:27:09.821248 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:09.821207    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:27:09.917468 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:09.917420    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:27:10.066428 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:10.066337    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:27:10.439874 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:10.439840    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:27:10.502540 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:10.502503    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:27:25.772633 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:25.772600    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-jmz4j"]
Jul 02 21:27:25.775713 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:25.775696    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jmz4j"
Jul 02 21:27:25.778572 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:25.778542    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 02 21:27:25.778572 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:25.778542    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 02 21:27:25.778746 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:25.778572    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 02 21:27:25.779677 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:25.779662    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9qnb9\""
Jul 02 21:27:25.784093 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:25.784073    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-jmz4j"]
Jul 02 21:27:25.821707 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:25.821686    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/264e233b-c7a9-4bea-ae95-ccb7cc00bde6-model-cache\") pod \"s3-init-jmz4j\" (UID: \"264e233b-c7a9-4bea-ae95-ccb7cc00bde6\") " pod="kserve/s3-init-jmz4j"
Jul 02 21:27:25.821795 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:25.821738    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l2rl\" (UniqueName: \"kubernetes.io/projected/264e233b-c7a9-4bea-ae95-ccb7cc00bde6-kube-api-access-9l2rl\") pod \"s3-init-jmz4j\" (UID: \"264e233b-c7a9-4bea-ae95-ccb7cc00bde6\") " pod="kserve/s3-init-jmz4j"
Jul 02 21:27:25.922323 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:25.922299    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l2rl\" (UniqueName: \"kubernetes.io/projected/264e233b-c7a9-4bea-ae95-ccb7cc00bde6-kube-api-access-9l2rl\") pod \"s3-init-jmz4j\" (UID: \"264e233b-c7a9-4bea-ae95-ccb7cc00bde6\") " pod="kserve/s3-init-jmz4j"
Jul 02 21:27:25.922421 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:25.922340    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/264e233b-c7a9-4bea-ae95-ccb7cc00bde6-model-cache\") pod \"s3-init-jmz4j\" (UID: \"264e233b-c7a9-4bea-ae95-ccb7cc00bde6\") " pod="kserve/s3-init-jmz4j"
Jul 02 21:27:25.922649 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:25.922634    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/264e233b-c7a9-4bea-ae95-ccb7cc00bde6-model-cache\") pod \"s3-init-jmz4j\" (UID: \"264e233b-c7a9-4bea-ae95-ccb7cc00bde6\") " pod="kserve/s3-init-jmz4j"
Jul 02 21:27:25.930914 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:25.930896    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l2rl\" (UniqueName: \"kubernetes.io/projected/264e233b-c7a9-4bea-ae95-ccb7cc00bde6-kube-api-access-9l2rl\") pod \"s3-init-jmz4j\" (UID: \"264e233b-c7a9-4bea-ae95-ccb7cc00bde6\") " pod="kserve/s3-init-jmz4j"
Jul 02 21:27:26.095958 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:26.095908    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jmz4j"
Jul 02 21:27:26.220556 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:26.220529    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-jmz4j"]
Jul 02 21:27:26.224103 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:27:26.224063    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod264e233b_c7a9_4bea_ae95_ccb7cc00bde6.slice/crio-6e0e7c41889cb951b0a6b9293437c911b2f5f45fc6ef0456323ef35fbb652f94 WatchSource:0}: Error finding container 6e0e7c41889cb951b0a6b9293437c911b2f5f45fc6ef0456323ef35fbb652f94: Status 404 returned error can't find the container with id 6e0e7c41889cb951b0a6b9293437c911b2f5f45fc6ef0456323ef35fbb652f94
Jul 02 21:27:26.225915 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:26.225894    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 21:27:27.099074 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:27.099038    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jmz4j" event={"ID":"264e233b-c7a9-4bea-ae95-ccb7cc00bde6","Type":"ContainerStarted","Data":"6e0e7c41889cb951b0a6b9293437c911b2f5f45fc6ef0456323ef35fbb652f94"}
Jul 02 21:27:28.390894 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:28.390859    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:27:28.446348 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:28.446320    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:27:30.112945 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:30.112908    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jmz4j" event={"ID":"264e233b-c7a9-4bea-ae95-ccb7cc00bde6","Type":"ContainerStarted","Data":"c797a1bc27759bbeb04b625b39a53ecbe30d44f045a5b48da5389f9628b6f170"}
Jul 02 21:27:32.121788 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:32.121756    2575 generic.go:356] "Generic (PLEG): container finished" podID="264e233b-c7a9-4bea-ae95-ccb7cc00bde6" containerID="c797a1bc27759bbeb04b625b39a53ecbe30d44f045a5b48da5389f9628b6f170" exitCode=0
Jul 02 21:27:32.122189 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:32.121836    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jmz4j" event={"ID":"264e233b-c7a9-4bea-ae95-ccb7cc00bde6","Type":"ContainerDied","Data":"c797a1bc27759bbeb04b625b39a53ecbe30d44f045a5b48da5389f9628b6f170"}
Jul 02 21:27:37.145495 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:37.145450    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jmz4j" event={"ID":"264e233b-c7a9-4bea-ae95-ccb7cc00bde6","Type":"ContainerStarted","Data":"e303e790647ddcc6d3d9f508dc9e539e7b4adbc9b39c342b134b081a5a337507"}
Jul 02 21:27:37.163605 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:27:37.163550    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-jmz4j" podStartSLOduration=2.6974835 podStartE2EDuration="12.163533315s" totalImagesPullingTime="7.869712833s" totalInitContainerRuntime="1.596336982s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:27:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:27:37.163101557 +0000 UTC m=+528.789603454" watchObservedRunningTime="2026-07-02 21:27:37.163533315 +0000 UTC m=+528.790035193"
Jul 02 21:28:42.395075 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:42.395040    2575 generic.go:356] "Generic (PLEG): container finished" podID="264e233b-c7a9-4bea-ae95-ccb7cc00bde6" containerID="e303e790647ddcc6d3d9f508dc9e539e7b4adbc9b39c342b134b081a5a337507" exitCode=0
Jul 02 21:28:42.395504 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:42.395107    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jmz4j" event={"ID":"264e233b-c7a9-4bea-ae95-ccb7cc00bde6","Type":"ContainerDied","Data":"e303e790647ddcc6d3d9f508dc9e539e7b4adbc9b39c342b134b081a5a337507"}
Jul 02 21:28:43.519456 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:43.519429    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jmz4j"
Jul 02 21:28:43.658357 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:43.658294    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9l2rl\" (UniqueName: \"kubernetes.io/projected/264e233b-c7a9-4bea-ae95-ccb7cc00bde6-kube-api-access-9l2rl\") pod \"264e233b-c7a9-4bea-ae95-ccb7cc00bde6\" (UID: \"264e233b-c7a9-4bea-ae95-ccb7cc00bde6\") "
Jul 02 21:28:43.658490 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:43.658407    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/264e233b-c7a9-4bea-ae95-ccb7cc00bde6-model-cache\") pod \"264e233b-c7a9-4bea-ae95-ccb7cc00bde6\" (UID: \"264e233b-c7a9-4bea-ae95-ccb7cc00bde6\") "
Jul 02 21:28:43.660389 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:43.660361    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264e233b-c7a9-4bea-ae95-ccb7cc00bde6-kube-api-access-9l2rl" (OuterVolumeSpecName: "kube-api-access-9l2rl") pod "264e233b-c7a9-4bea-ae95-ccb7cc00bde6" (UID: "264e233b-c7a9-4bea-ae95-ccb7cc00bde6"). InnerVolumeSpecName "kube-api-access-9l2rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:28:43.676899 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:43.676870    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264e233b-c7a9-4bea-ae95-ccb7cc00bde6-model-cache" (OuterVolumeSpecName: "model-cache") pod "264e233b-c7a9-4bea-ae95-ccb7cc00bde6" (UID: "264e233b-c7a9-4bea-ae95-ccb7cc00bde6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:28:43.759283 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:43.759263    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/264e233b-c7a9-4bea-ae95-ccb7cc00bde6-model-cache\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:28:43.759283 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:43.759283    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l2rl\" (UniqueName: \"kubernetes.io/projected/264e233b-c7a9-4bea-ae95-ccb7cc00bde6-kube-api-access-9l2rl\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:28:44.404056 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:44.404028    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jmz4j"
Jul 02 21:28:44.404237 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:44.404028    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jmz4j" event={"ID":"264e233b-c7a9-4bea-ae95-ccb7cc00bde6","Type":"ContainerDied","Data":"6e0e7c41889cb951b0a6b9293437c911b2f5f45fc6ef0456323ef35fbb652f94"}
Jul 02 21:28:44.404237 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:44.404144    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e0e7c41889cb951b0a6b9293437c911b2f5f45fc6ef0456323ef35fbb652f94"
Jul 02 21:28:44.771748 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:44.771680    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:28:44.872178 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:44.872153    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:28:56.678154 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:56.678117    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:28:56.811450 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:56.811414    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"]
Jul 02 21:28:56.811722 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:56.811710    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="264e233b-c7a9-4bea-ae95-ccb7cc00bde6" containerName="download-hf-model"
Jul 02 21:28:56.811765 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:56.811723    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="264e233b-c7a9-4bea-ae95-ccb7cc00bde6" containerName="download-hf-model"
Jul 02 21:28:56.811765 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:56.811742    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="264e233b-c7a9-4bea-ae95-ccb7cc00bde6" containerName="s3-init"
Jul 02 21:28:56.811765 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:56.811748    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="264e233b-c7a9-4bea-ae95-ccb7cc00bde6" containerName="s3-init"
Jul 02 21:28:56.811856 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:56.811801    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="264e233b-c7a9-4bea-ae95-ccb7cc00bde6" containerName="s3-init"
Jul 02 21:28:56.815349 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:56.815334    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:28:56.818142 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:56.818113    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-ed912-kube-rbac-proxy-sar-config\""
Jul 02 21:28:56.818142 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:56.818133    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 21:28:56.818337 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:56.818152    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 21:28:56.818504 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:56.818481    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 21:28:56.818594 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:56.818487    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-ed912-predictor-serving-cert\""
Jul 02 21:28:56.819255 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:56.819237    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 21:28:56.819359 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:56.819342    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-57xll\""
Jul 02 21:28:56.833397 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:56.833379    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"]
Jul 02 21:28:56.841658 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:56.841636    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:28:56.960379 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:56.960327    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a4bb54c-cae0-4723-ad7c-aca4f345044e-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn\" (UID: \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:28:56.960379 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:56.960354    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a4bb54c-cae0-4723-ad7c-aca4f345044e-proxy-tls\") pod \"isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn\" (UID: \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:28:56.960542 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:56.960394    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a4bb54c-cae0-4723-ad7c-aca4f345044e-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn\" (UID: \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:28:56.960542 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:56.960483    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a4bb54c-cae0-4723-ad7c-aca4f345044e-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn\" (UID: \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:28:56.960629 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:56.960538    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz9h8\" (UniqueName: \"kubernetes.io/projected/3a4bb54c-cae0-4723-ad7c-aca4f345044e-kube-api-access-gz9h8\") pod \"isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn\" (UID: \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:28:57.061681 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:57.061657    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a4bb54c-cae0-4723-ad7c-aca4f345044e-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn\" (UID: \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:28:57.061792 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:57.061701    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a4bb54c-cae0-4723-ad7c-aca4f345044e-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn\" (UID: \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:28:57.061792 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:57.061740    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz9h8\" (UniqueName: \"kubernetes.io/projected/3a4bb54c-cae0-4723-ad7c-aca4f345044e-kube-api-access-gz9h8\") pod \"isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn\" (UID: \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:28:57.061792 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:57.061778    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a4bb54c-cae0-4723-ad7c-aca4f345044e-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn\" (UID: \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:28:57.061957 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:57.061805    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a4bb54c-cae0-4723-ad7c-aca4f345044e-proxy-tls\") pod \"isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn\" (UID: \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:28:57.062076 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:57.062049    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a4bb54c-cae0-4723-ad7c-aca4f345044e-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn\" (UID: \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:28:57.062331 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:57.062313    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a4bb54c-cae0-4723-ad7c-aca4f345044e-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn\" (UID: \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:28:57.062401 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:57.062358    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a4bb54c-cae0-4723-ad7c-aca4f345044e-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn\" (UID: \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:28:57.064098 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:57.064078    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a4bb54c-cae0-4723-ad7c-aca4f345044e-proxy-tls\") pod \"isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn\" (UID: \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:28:57.071131 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:57.071109    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz9h8\" (UniqueName: \"kubernetes.io/projected/3a4bb54c-cae0-4723-ad7c-aca4f345044e-kube-api-access-gz9h8\") pod \"isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn\" (UID: \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:28:57.124955 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:57.124925    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:28:57.252261 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:57.252191    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"]
Jul 02 21:28:57.255252 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:28:57.255221    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a4bb54c_cae0_4723_ad7c_aca4f345044e.slice/crio-45a380f4c41c4cef2b3d709682b518692ed586dc26e47f65f5139b1dd120631c WatchSource:0}: Error finding container 45a380f4c41c4cef2b3d709682b518692ed586dc26e47f65f5139b1dd120631c: Status 404 returned error can't find the container with id 45a380f4c41c4cef2b3d709682b518692ed586dc26e47f65f5139b1dd120631c
Jul 02 21:28:57.455829 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:57.455797    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" event={"ID":"3a4bb54c-cae0-4723-ad7c-aca4f345044e","Type":"ContainerStarted","Data":"4db1f7ccbe168d4045ac03787bba64bc664433d336be387e3957396e6ac0c186"}
Jul 02 21:28:57.455998 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:28:57.455859    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" event={"ID":"3a4bb54c-cae0-4723-ad7c-aca4f345044e","Type":"ContainerStarted","Data":"45a380f4c41c4cef2b3d709682b518692ed586dc26e47f65f5139b1dd120631c"}
Jul 02 21:29:01.472919 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:01.472886    2575 generic.go:356] "Generic (PLEG): container finished" podID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerID="4db1f7ccbe168d4045ac03787bba64bc664433d336be387e3957396e6ac0c186" exitCode=0
Jul 02 21:29:01.473332 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:01.472930    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" event={"ID":"3a4bb54c-cae0-4723-ad7c-aca4f345044e","Type":"ContainerDied","Data":"4db1f7ccbe168d4045ac03787bba64bc664433d336be387e3957396e6ac0c186"}
Jul 02 21:29:16.550187 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:16.550139    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" event={"ID":"3a4bb54c-cae0-4723-ad7c-aca4f345044e","Type":"ContainerStarted","Data":"d486e917f81e1f836b1c212a57ae41550e579667be27b83249e91899f1ffcc56"}
Jul 02 21:29:19.565533 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:19.565500    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" event={"ID":"3a4bb54c-cae0-4723-ad7c-aca4f345044e","Type":"ContainerStarted","Data":"2906b42be4dd68ce8b2251e29749757db44ab9e8f74bba3e33d4fb1d0e2831af"}
Jul 02 21:29:21.575962 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:21.575930    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" event={"ID":"3a4bb54c-cae0-4723-ad7c-aca4f345044e","Type":"ContainerStarted","Data":"f47648de76747d96b8cfffdc851baf8c85457e53d53d7b024e33556e69f75bda"}
Jul 02 21:29:21.576379 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:21.576025    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:29:21.576379 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:21.576152    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:29:21.577462 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:21.577437    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 21:29:21.598340 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:21.598297    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podStartSLOduration=2.032515049 podStartE2EDuration="25.598283916s" totalImagesPullingTime="19.759213732s" totalInitContainerRuntime="3.806555135s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:28:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:29:21.595966107 +0000 UTC m=+633.222467985" watchObservedRunningTime="2026-07-02 21:29:21.598283916 +0000 UTC m=+633.224785791"
Jul 02 21:29:22.579894 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:22.579862    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:29:22.580387 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:22.580033    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 21:29:22.581091 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:22.581064    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:29:23.583978 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:23.583935    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 21:29:23.584444 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:23.584305    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:29:23.587928 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:23.587908    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:29:24.587490 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:24.587453    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 21:29:24.587908 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:24.587744    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:29:28.442326 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:28.442296    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:29:29.649877 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:29.649845    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:29:33.291985 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:33.291953    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:29:34.588227 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:34.588155    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 21:29:34.588619 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:34.588560    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:29:44.587686 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:44.587642    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 21:29:44.588115 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:44.588090    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:29:54.587825 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:54.587777    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 21:29:54.588290 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:29:54.588266    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:30:04.587371 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:04.587327    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 21:30:04.587827 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:04.587695    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:30:09.908566 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:09.908510    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:30:14.587534 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:14.587490    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 21:30:14.588019 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:14.587994    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:30:24.587939 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:24.587893    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 21:30:24.588449 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:24.588424    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:30:25.182306 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:25.182267    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:30:25.240233 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:25.240202    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:30:27.009800 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:27.009742    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 21:30:27.010219 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:27.010198    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:30:37.014570 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:37.014541    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:30:37.015047 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:37.014633    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:30:51.788157 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:51.788124    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:30:51.845868 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:51.845837    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"]
Jul 02 21:30:51.846224 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:51.846187    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kube-rbac-proxy" containerID="cri-o://2906b42be4dd68ce8b2251e29749757db44ab9e8f74bba3e33d4fb1d0e2831af" gracePeriod=30
Jul 02 21:30:51.846373 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:51.846209    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="agent" containerID="cri-o://f47648de76747d96b8cfffdc851baf8c85457e53d53d7b024e33556e69f75bda" gracePeriod=30
Jul 02 21:30:51.846373 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:51.846145    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kserve-container" containerID="cri-o://d486e917f81e1f836b1c212a57ae41550e579667be27b83249e91899f1ffcc56" gracePeriod=30
Jul 02 21:30:51.861157 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:51.861126    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"]
Jul 02 21:30:51.864352 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:51.864331    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:30:51.867026 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:51.867004    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-c1b1e-kube-rbac-proxy-sar-config\""
Jul 02 21:30:51.867138 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:51.867026    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-c1b1e-predictor-serving-cert\""
Jul 02 21:30:51.878868 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:51.878798    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"]
Jul 02 21:30:51.891408 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:51.891365    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:30:51.952472 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:51.952441    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"]
Jul 02 21:30:51.955099 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:51.955079    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:30:51.958149 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:51.958127    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-c1b1e-predictor-serving-cert\""
Jul 02 21:30:51.958295 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:51.958241    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-c1b1e-kube-rbac-proxy-sar-config\""
Jul 02 21:30:51.970453 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:51.970432    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"]
Jul 02 21:30:51.994193 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:51.993899    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:30:52.039853 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.039761    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa460ab6-5679-4644-8562-424253a015c3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn\" (UID: \"fa460ab6-5679-4644-8562-424253a015c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:30:52.039853 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.039804    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa460ab6-5679-4644-8562-424253a015c3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn\" (UID: \"fa460ab6-5679-4644-8562-424253a015c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:30:52.040078 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.039887    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkfn6\" (UniqueName: \"kubernetes.io/projected/fa460ab6-5679-4644-8562-424253a015c3-kube-api-access-hkfn6\") pod \"isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn\" (UID: \"fa460ab6-5679-4644-8562-424253a015c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:30:52.040078 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.039922    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa460ab6-5679-4644-8562-424253a015c3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn\" (UID: \"fa460ab6-5679-4644-8562-424253a015c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:30:52.040078 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.039991    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa460ab6-5679-4644-8562-424253a015c3-proxy-tls\") pod \"isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn\" (UID: \"fa460ab6-5679-4644-8562-424253a015c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:30:52.090067 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.090033    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:30:52.141398 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.141372    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ht98\" (UniqueName: \"kubernetes.io/projected/f252efa2-359f-4eb0-8a51-c2cf43f952af-kube-api-access-2ht98\") pod \"isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw\" (UID: \"f252efa2-359f-4eb0-8a51-c2cf43f952af\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:30:52.141546 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.141413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa460ab6-5679-4644-8562-424253a015c3-proxy-tls\") pod \"isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn\" (UID: \"fa460ab6-5679-4644-8562-424253a015c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:30:52.141546 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.141458    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f252efa2-359f-4eb0-8a51-c2cf43f952af-cabundle-cert\") pod \"isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw\" (UID: \"f252efa2-359f-4eb0-8a51-c2cf43f952af\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:30:52.141546 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.141493    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f252efa2-359f-4eb0-8a51-c2cf43f952af-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw\" (UID: \"f252efa2-359f-4eb0-8a51-c2cf43f952af\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:30:52.141711 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.141592    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa460ab6-5679-4644-8562-424253a015c3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn\" (UID: \"fa460ab6-5679-4644-8562-424253a015c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:30:52.141711 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.141636    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa460ab6-5679-4644-8562-424253a015c3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn\" (UID: \"fa460ab6-5679-4644-8562-424253a015c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:30:52.141711 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.141673    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f252efa2-359f-4eb0-8a51-c2cf43f952af-proxy-tls\") pod \"isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw\" (UID: \"f252efa2-359f-4eb0-8a51-c2cf43f952af\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:30:52.141864 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.141731    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkfn6\" (UniqueName: \"kubernetes.io/projected/fa460ab6-5679-4644-8562-424253a015c3-kube-api-access-hkfn6\") pod \"isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn\" (UID: \"fa460ab6-5679-4644-8562-424253a015c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:30:52.141864 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.141765    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f252efa2-359f-4eb0-8a51-c2cf43f952af-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw\" (UID: \"f252efa2-359f-4eb0-8a51-c2cf43f952af\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:30:52.141864 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.141805    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa460ab6-5679-4644-8562-424253a015c3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn\" (UID: \"fa460ab6-5679-4644-8562-424253a015c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:30:52.142204 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.142185    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa460ab6-5679-4644-8562-424253a015c3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn\" (UID: \"fa460ab6-5679-4644-8562-424253a015c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:30:52.142302 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.142286    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa460ab6-5679-4644-8562-424253a015c3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn\" (UID: \"fa460ab6-5679-4644-8562-424253a015c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:30:52.142705 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.142683    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa460ab6-5679-4644-8562-424253a015c3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn\" (UID: \"fa460ab6-5679-4644-8562-424253a015c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:30:52.143821 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.143803    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa460ab6-5679-4644-8562-424253a015c3-proxy-tls\") pod \"isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn\" (UID: \"fa460ab6-5679-4644-8562-424253a015c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:30:52.149397 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.149378    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkfn6\" (UniqueName: \"kubernetes.io/projected/fa460ab6-5679-4644-8562-424253a015c3-kube-api-access-hkfn6\") pod \"isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn\" (UID: \"fa460ab6-5679-4644-8562-424253a015c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:30:52.176993 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.176968    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:30:52.218036 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.218006    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:30:52.242627 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.242600    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f252efa2-359f-4eb0-8a51-c2cf43f952af-cabundle-cert\") pod \"isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw\" (UID: \"f252efa2-359f-4eb0-8a51-c2cf43f952af\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:30:52.242740 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.242637    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f252efa2-359f-4eb0-8a51-c2cf43f952af-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw\" (UID: \"f252efa2-359f-4eb0-8a51-c2cf43f952af\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:30:52.242740 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.242673    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f252efa2-359f-4eb0-8a51-c2cf43f952af-proxy-tls\") pod \"isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw\" (UID: \"f252efa2-359f-4eb0-8a51-c2cf43f952af\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:30:52.242740 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.242709    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f252efa2-359f-4eb0-8a51-c2cf43f952af-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw\" (UID: \"f252efa2-359f-4eb0-8a51-c2cf43f952af\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:30:52.242740 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.242737    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ht98\" (UniqueName: \"kubernetes.io/projected/f252efa2-359f-4eb0-8a51-c2cf43f952af-kube-api-access-2ht98\") pod \"isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw\" (UID: \"f252efa2-359f-4eb0-8a51-c2cf43f952af\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:30:52.243231 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.243208    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f252efa2-359f-4eb0-8a51-c2cf43f952af-cabundle-cert\") pod \"isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw\" (UID: \"f252efa2-359f-4eb0-8a51-c2cf43f952af\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:30:52.243456 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.243419    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f252efa2-359f-4eb0-8a51-c2cf43f952af-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw\" (UID: \"f252efa2-359f-4eb0-8a51-c2cf43f952af\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:30:52.243541 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.243486    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f252efa2-359f-4eb0-8a51-c2cf43f952af-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw\" (UID: \"f252efa2-359f-4eb0-8a51-c2cf43f952af\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:30:52.245566 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.245545    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f252efa2-359f-4eb0-8a51-c2cf43f952af-proxy-tls\") pod \"isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw\" (UID: \"f252efa2-359f-4eb0-8a51-c2cf43f952af\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:30:52.251439 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.251411    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ht98\" (UniqueName: \"kubernetes.io/projected/f252efa2-359f-4eb0-8a51-c2cf43f952af-kube-api-access-2ht98\") pod \"isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw\" (UID: \"f252efa2-359f-4eb0-8a51-c2cf43f952af\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:30:52.265576 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.265550    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:30:52.309065 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.309040    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"]
Jul 02 21:30:52.310354 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:30:52.310331    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa460ab6_5679_4644_8562_424253a015c3.slice/crio-95b66eb627e99ca841c4deb7dab8d44064505ceff181b304ec086baa3c0f3a8c WatchSource:0}: Error finding container 95b66eb627e99ca841c4deb7dab8d44064505ceff181b304ec086baa3c0f3a8c: Status 404 returned error can't find the container with id 95b66eb627e99ca841c4deb7dab8d44064505ceff181b304ec086baa3c0f3a8c
Jul 02 21:30:52.400701 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.400680    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"]
Jul 02 21:30:52.403558 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:30:52.403529    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf252efa2_359f_4eb0_8a51_c2cf43f952af.slice/crio-ade2ac12a7651f52bfcd288c57dcad7d7c772095a0c039b9773b9e168e76f8e5 WatchSource:0}: Error finding container ade2ac12a7651f52bfcd288c57dcad7d7c772095a0c039b9773b9e168e76f8e5: Status 404 returned error can't find the container with id ade2ac12a7651f52bfcd288c57dcad7d7c772095a0c039b9773b9e168e76f8e5
Jul 02 21:30:52.941354 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.941318    2575 generic.go:356] "Generic (PLEG): container finished" podID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerID="2906b42be4dd68ce8b2251e29749757db44ab9e8f74bba3e33d4fb1d0e2831af" exitCode=2
Jul 02 21:30:52.941749 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.941386    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" event={"ID":"3a4bb54c-cae0-4723-ad7c-aca4f345044e","Type":"ContainerDied","Data":"2906b42be4dd68ce8b2251e29749757db44ab9e8f74bba3e33d4fb1d0e2831af"}
Jul 02 21:30:52.943024 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.943001    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw" event={"ID":"f252efa2-359f-4eb0-8a51-c2cf43f952af","Type":"ContainerStarted","Data":"d98b7918781cb34e0328620b80664d5822f9af13344766762f5d3e269cd50460"}
Jul 02 21:30:52.943147 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.943055    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw" event={"ID":"f252efa2-359f-4eb0-8a51-c2cf43f952af","Type":"ContainerStarted","Data":"ade2ac12a7651f52bfcd288c57dcad7d7c772095a0c039b9773b9e168e76f8e5"}
Jul 02 21:30:52.944472 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.944444    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" event={"ID":"fa460ab6-5679-4644-8562-424253a015c3","Type":"ContainerStarted","Data":"7503c427c9369b3598085508b8869826196307e9981d2559a2c54443747417df"}
Jul 02 21:30:52.944554 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:52.944501    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" event={"ID":"fa460ab6-5679-4644-8562-424253a015c3","Type":"ContainerStarted","Data":"95b66eb627e99ca841c4deb7dab8d44064505ceff181b304ec086baa3c0f3a8c"}
Jul 02 21:30:53.584596 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:53.584554    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 02 21:30:55.958691 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:55.958662    2575 generic.go:356] "Generic (PLEG): container finished" podID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerID="d98b7918781cb34e0328620b80664d5822f9af13344766762f5d3e269cd50460" exitCode=0
Jul 02 21:30:55.959016 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:55.958756    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw" event={"ID":"f252efa2-359f-4eb0-8a51-c2cf43f952af","Type":"ContainerDied","Data":"d98b7918781cb34e0328620b80664d5822f9af13344766762f5d3e269cd50460"}
Jul 02 21:30:55.960522 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:55.960496    2575 generic.go:356] "Generic (PLEG): container finished" podID="fa460ab6-5679-4644-8562-424253a015c3" containerID="7503c427c9369b3598085508b8869826196307e9981d2559a2c54443747417df" exitCode=0
Jul 02 21:30:55.960646 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:55.960557    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" event={"ID":"fa460ab6-5679-4644-8562-424253a015c3","Type":"ContainerDied","Data":"7503c427c9369b3598085508b8869826196307e9981d2559a2c54443747417df"}
Jul 02 21:30:56.974576 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:56.974532    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" event={"ID":"fa460ab6-5679-4644-8562-424253a015c3","Type":"ContainerStarted","Data":"66f3449808eff78bb56ccf828a053a6b293db121540f98b596d9f7185ee316d5"}
Jul 02 21:30:56.975067 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:56.974686    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" event={"ID":"fa460ab6-5679-4644-8562-424253a015c3","Type":"ContainerStarted","Data":"4d999282a0f644a5113ee038ee30cc3b5ec750cd64d9dc262e888c1b093ba902"}
Jul 02 21:30:56.975151 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:56.975070    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:30:56.975151 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:56.975140    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:30:56.976767 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:56.976734    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" podUID="fa460ab6-5679-4644-8562-424253a015c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 02 21:30:56.977994 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:56.977955    2575 generic.go:356] "Generic (PLEG): container finished" podID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerID="d486e917f81e1f836b1c212a57ae41550e579667be27b83249e91899f1ffcc56" exitCode=0
Jul 02 21:30:56.978114 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:56.978012    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" event={"ID":"3a4bb54c-cae0-4723-ad7c-aca4f345044e","Type":"ContainerDied","Data":"d486e917f81e1f836b1c212a57ae41550e579667be27b83249e91899f1ffcc56"}
Jul 02 21:30:56.996695 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:56.996562    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" podStartSLOduration=2.443579428 podStartE2EDuration="5.996545558s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.55296613s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:30:56.994827079 +0000 UTC m=+728.621328978" watchObservedRunningTime="2026-07-02 21:30:56.996545558 +0000 UTC m=+728.623047435"
Jul 02 21:30:57.011032 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:57.010068    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 21:30:57.011032 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:57.010725    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:30:57.985112 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:57.985070    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" podUID="fa460ab6-5679-4644-8562-424253a015c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 02 21:30:58.584459 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:30:58.584415    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 02 21:31:02.990925 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:02.990896    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:31:02.991602 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:02.991570    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" podUID="fa460ab6-5679-4644-8562-424253a015c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 02 21:31:03.584395 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:03.584355    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 02 21:31:03.584603 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:03.584501    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:31:07.009807 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:07.009766    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 21:31:07.010287 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:07.010067    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:31:08.584234 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:08.584186    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 02 21:31:12.991909 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:12.991864    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" podUID="fa460ab6-5679-4644-8562-424253a015c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 02 21:31:13.584449 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:13.584409    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 02 21:31:16.072273 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:16.072192    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw" event={"ID":"f252efa2-359f-4eb0-8a51-c2cf43f952af","Type":"ContainerStarted","Data":"2520263b1b105194be4c9e96935caf5338551bdb90f49eded0c19b605688e54a"}
Jul 02 21:31:16.072702 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:16.072297    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw" event={"ID":"f252efa2-359f-4eb0-8a51-c2cf43f952af","Type":"ContainerStarted","Data":"7906be70aed2eb27bfa498a8db1d4966131d1ebe9bc707028696a68cdbb247df"}
Jul 02 21:31:16.072702 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:16.072517    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:31:16.095302 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:16.095256    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw" podStartSLOduration=1.883534389 podStartE2EDuration="25.09524148s" totalImagesPullingTime="19.770780214s" totalInitContainerRuntime="3.440926877s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:31:16.093976624 +0000 UTC m=+747.720478502" watchObservedRunningTime="2026-07-02 21:31:16.09524148 +0000 UTC m=+747.721743356"
Jul 02 21:31:17.009723 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:17.009678    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 02 21:31:17.010047 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:17.010020    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:31:17.013474 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:17.013454    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:31:17.013566 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:17.013550    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:31:17.076181 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:17.076145    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:31:17.077249 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:17.077206    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw" podUID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 02 21:31:18.080142 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:18.080089    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw" podUID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 02 21:31:18.584082 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:18.584045    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 02 21:31:22.524409 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:22.524380    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:31:22.608573 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:22.608545    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a4bb54c-cae0-4723-ad7c-aca4f345044e-kube-rbac-proxy-sar-config\") pod \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\" (UID: \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\") "
Jul 02 21:31:22.608723 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:22.608584    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a4bb54c-cae0-4723-ad7c-aca4f345044e-kserve-provision-location\") pod \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\" (UID: \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\") "
Jul 02 21:31:22.608723 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:22.608605    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a4bb54c-cae0-4723-ad7c-aca4f345044e-proxy-tls\") pod \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\" (UID: \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\") "
Jul 02 21:31:22.608723 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:22.608623    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz9h8\" (UniqueName: \"kubernetes.io/projected/3a4bb54c-cae0-4723-ad7c-aca4f345044e-kube-api-access-gz9h8\") pod \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\" (UID: \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\") "
Jul 02 21:31:22.608886 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:22.608764    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a4bb54c-cae0-4723-ad7c-aca4f345044e-cabundle-cert\") pod \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\" (UID: \"3a4bb54c-cae0-4723-ad7c-aca4f345044e\") "
Jul 02 21:31:22.608886 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:22.608849    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4bb54c-cae0-4723-ad7c-aca4f345044e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3a4bb54c-cae0-4723-ad7c-aca4f345044e" (UID: "3a4bb54c-cae0-4723-ad7c-aca4f345044e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:31:22.608982 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:22.608935    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a4bb54c-cae0-4723-ad7c-aca4f345044e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a4bb54c-cae0-4723-ad7c-aca4f345044e" (UID: "3a4bb54c-cae0-4723-ad7c-aca4f345044e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:31:22.609099 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:22.609077    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a4bb54c-cae0-4723-ad7c-aca4f345044e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3a4bb54c-cae0-4723-ad7c-aca4f345044e" (UID: "3a4bb54c-cae0-4723-ad7c-aca4f345044e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:31:22.609161 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:22.609138    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a4bb54c-cae0-4723-ad7c-aca4f345044e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:31:22.609161 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:22.609151    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a4bb54c-cae0-4723-ad7c-aca4f345044e-kserve-provision-location\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:31:22.609283 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:22.609188    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a4bb54c-cae0-4723-ad7c-aca4f345044e-cabundle-cert\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:31:22.610823 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:22.610800    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a4bb54c-cae0-4723-ad7c-aca4f345044e-kube-api-access-gz9h8" (OuterVolumeSpecName: "kube-api-access-gz9h8") pod "3a4bb54c-cae0-4723-ad7c-aca4f345044e" (UID: "3a4bb54c-cae0-4723-ad7c-aca4f345044e"). InnerVolumeSpecName "kube-api-access-gz9h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:31:22.610926 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:22.610841    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a4bb54c-cae0-4723-ad7c-aca4f345044e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a4bb54c-cae0-4723-ad7c-aca4f345044e" (UID: "3a4bb54c-cae0-4723-ad7c-aca4f345044e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:31:22.710525 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:22.710461    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a4bb54c-cae0-4723-ad7c-aca4f345044e-proxy-tls\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:31:22.710525 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:22.710487    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz9h8\" (UniqueName: \"kubernetes.io/projected/3a4bb54c-cae0-4723-ad7c-aca4f345044e-kube-api-access-gz9h8\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:31:22.991846 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:22.991769    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" podUID="fa460ab6-5679-4644-8562-424253a015c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 02 21:31:23.083915 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:23.083889    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:31:23.084319 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:23.084286    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw" podUID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 02 21:31:23.101309 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:23.101284    2575 generic.go:356] "Generic (PLEG): container finished" podID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerID="f47648de76747d96b8cfffdc851baf8c85457e53d53d7b024e33556e69f75bda" exitCode=0
Jul 02 21:31:23.101411 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:23.101338    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"
Jul 02 21:31:23.101411 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:23.101394    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" event={"ID":"3a4bb54c-cae0-4723-ad7c-aca4f345044e","Type":"ContainerDied","Data":"f47648de76747d96b8cfffdc851baf8c85457e53d53d7b024e33556e69f75bda"}
Jul 02 21:31:23.101487 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:23.101455    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn" event={"ID":"3a4bb54c-cae0-4723-ad7c-aca4f345044e","Type":"ContainerDied","Data":"45a380f4c41c4cef2b3d709682b518692ed586dc26e47f65f5139b1dd120631c"}
Jul 02 21:31:23.101521 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:23.101501    2575 scope.go:117] "RemoveContainer" containerID="f47648de76747d96b8cfffdc851baf8c85457e53d53d7b024e33556e69f75bda"
Jul 02 21:31:23.110097 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:23.110055    2575 scope.go:117] "RemoveContainer" containerID="2906b42be4dd68ce8b2251e29749757db44ab9e8f74bba3e33d4fb1d0e2831af"
Jul 02 21:31:23.117483 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:23.117465    2575 scope.go:117] "RemoveContainer" containerID="d486e917f81e1f836b1c212a57ae41550e579667be27b83249e91899f1ffcc56"
Jul 02 21:31:23.120723 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:23.120703    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"]
Jul 02 21:31:23.125126 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:23.125105    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ed912-predictor-98d8544d9-h4kfn"]
Jul 02 21:31:23.125824 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:23.125808    2575 scope.go:117] "RemoveContainer" containerID="4db1f7ccbe168d4045ac03787bba64bc664433d336be387e3957396e6ac0c186"
Jul 02 21:31:23.132712 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:23.132692    2575 scope.go:117] "RemoveContainer" containerID="f47648de76747d96b8cfffdc851baf8c85457e53d53d7b024e33556e69f75bda"
Jul 02 21:31:23.132964 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:31:23.132947    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f47648de76747d96b8cfffdc851baf8c85457e53d53d7b024e33556e69f75bda\": container with ID starting with f47648de76747d96b8cfffdc851baf8c85457e53d53d7b024e33556e69f75bda not found: ID does not exist" containerID="f47648de76747d96b8cfffdc851baf8c85457e53d53d7b024e33556e69f75bda"
Jul 02 21:31:23.133011 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:23.132972    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f47648de76747d96b8cfffdc851baf8c85457e53d53d7b024e33556e69f75bda"} err="failed to get container status \"f47648de76747d96b8cfffdc851baf8c85457e53d53d7b024e33556e69f75bda\": rpc error: code = NotFound desc = could not find container \"f47648de76747d96b8cfffdc851baf8c85457e53d53d7b024e33556e69f75bda\": container with ID starting with f47648de76747d96b8cfffdc851baf8c85457e53d53d7b024e33556e69f75bda not found: ID does not exist"
Jul 02 21:31:23.133011 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:23.132990    2575 scope.go:117] "RemoveContainer" containerID="2906b42be4dd68ce8b2251e29749757db44ab9e8f74bba3e33d4fb1d0e2831af"
Jul 02 21:31:23.133241 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:31:23.133223    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2906b42be4dd68ce8b2251e29749757db44ab9e8f74bba3e33d4fb1d0e2831af\": container with ID starting with 2906b42be4dd68ce8b2251e29749757db44ab9e8f74bba3e33d4fb1d0e2831af not found: ID does not exist" containerID="2906b42be4dd68ce8b2251e29749757db44ab9e8f74bba3e33d4fb1d0e2831af"
Jul 02 21:31:23.133290 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:23.133247    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2906b42be4dd68ce8b2251e29749757db44ab9e8f74bba3e33d4fb1d0e2831af"} err="failed to get container status \"2906b42be4dd68ce8b2251e29749757db44ab9e8f74bba3e33d4fb1d0e2831af\": rpc error: code = NotFound desc = could not find container \"2906b42be4dd68ce8b2251e29749757db44ab9e8f74bba3e33d4fb1d0e2831af\": container with ID starting with 2906b42be4dd68ce8b2251e29749757db44ab9e8f74bba3e33d4fb1d0e2831af not found: ID does not exist"
Jul 02 21:31:23.133290 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:23.133263    2575 scope.go:117] "RemoveContainer" containerID="d486e917f81e1f836b1c212a57ae41550e579667be27b83249e91899f1ffcc56"
Jul 02 21:31:23.133512 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:31:23.133487    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d486e917f81e1f836b1c212a57ae41550e579667be27b83249e91899f1ffcc56\": container with ID starting with d486e917f81e1f836b1c212a57ae41550e579667be27b83249e91899f1ffcc56 not found: ID does not exist" containerID="d486e917f81e1f836b1c212a57ae41550e579667be27b83249e91899f1ffcc56"
Jul 02 21:31:23.133553 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:23.133510    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d486e917f81e1f836b1c212a57ae41550e579667be27b83249e91899f1ffcc56"} err="failed to get container status \"d486e917f81e1f836b1c212a57ae41550e579667be27b83249e91899f1ffcc56\": rpc error: code = NotFound desc = could not find container \"d486e917f81e1f836b1c212a57ae41550e579667be27b83249e91899f1ffcc56\": container with ID starting with d486e917f81e1f836b1c212a57ae41550e579667be27b83249e91899f1ffcc56 not found: ID does not exist"
Jul 02 21:31:23.133553 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:23.133526    2575 scope.go:117] "RemoveContainer" containerID="4db1f7ccbe168d4045ac03787bba64bc664433d336be387e3957396e6ac0c186"
Jul 02 21:31:23.133755 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:31:23.133739    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db1f7ccbe168d4045ac03787bba64bc664433d336be387e3957396e6ac0c186\": container with ID starting with 4db1f7ccbe168d4045ac03787bba64bc664433d336be387e3957396e6ac0c186 not found: ID does not exist" containerID="4db1f7ccbe168d4045ac03787bba64bc664433d336be387e3957396e6ac0c186"
Jul 02 21:31:23.133797 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:23.133759    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db1f7ccbe168d4045ac03787bba64bc664433d336be387e3957396e6ac0c186"} err="failed to get container status \"4db1f7ccbe168d4045ac03787bba64bc664433d336be387e3957396e6ac0c186\": rpc error: code = NotFound desc = could not find container \"4db1f7ccbe168d4045ac03787bba64bc664433d336be387e3957396e6ac0c186\": container with ID starting with 4db1f7ccbe168d4045ac03787bba64bc664433d336be387e3957396e6ac0c186 not found: ID does not exist"
Jul 02 21:31:25.014447 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:25.014417    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" path="/var/lib/kubelet/pods/3a4bb54c-cae0-4723-ad7c-aca4f345044e/volumes"
Jul 02 21:31:32.992127 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:32.992086    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" podUID="fa460ab6-5679-4644-8562-424253a015c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 02 21:31:33.084377 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:33.084341    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw" podUID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 02 21:31:42.992123 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:42.992083    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" podUID="fa460ab6-5679-4644-8562-424253a015c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 02 21:31:43.085265 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:43.085224    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw" podUID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 02 21:31:52.992180 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:52.992137    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" podUID="fa460ab6-5679-4644-8562-424253a015c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 02 21:31:53.084334 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:31:53.084298    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw" podUID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 02 21:32:02.991510 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:02.991468    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" podUID="fa460ab6-5679-4644-8562-424253a015c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 02 21:32:03.085256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:03.085212    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw" podUID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 02 21:32:06.009220 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:06.009160    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" podUID="fa460ab6-5679-4644-8562-424253a015c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 02 21:32:13.085311 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:13.085218    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw" podUID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 02 21:32:16.009826 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:16.009794    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:32:23.085339 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:23.085305    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:32:31.916354 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:31.916314    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:32:32.243213 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:32.243129    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:32:41.976265 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:41.976229    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:32:42.146291 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.146261    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:32:42.221380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.221349    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"]
Jul 02 21:32:42.221778 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.221764    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="storage-initializer"
Jul 02 21:32:42.221778 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.221779    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="storage-initializer"
Jul 02 21:32:42.221861 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.221795    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="agent"
Jul 02 21:32:42.221861 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.221801    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="agent"
Jul 02 21:32:42.221861 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.221814    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kserve-container"
Jul 02 21:32:42.221861 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.221820    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kserve-container"
Jul 02 21:32:42.221861 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.221833    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kube-rbac-proxy"
Jul 02 21:32:42.221861 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.221839    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kube-rbac-proxy"
Jul 02 21:32:42.222100 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.221902    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kube-rbac-proxy"
Jul 02 21:32:42.222100 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.221913    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="agent"
Jul 02 21:32:42.222100 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.221922    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a4bb54c-cae0-4723-ad7c-aca4f345044e" containerName="kserve-container"
Jul 02 21:32:42.224608 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.224586    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:32:42.227444 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.227378    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-af6d2-kube-rbac-proxy-sar-config\""
Jul 02 21:32:42.227673 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.227654    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-af6d2-predictor-serving-cert\""
Jul 02 21:32:42.236446 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.236425    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"]
Jul 02 21:32:42.237437 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.237411    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" podUID="fa460ab6-5679-4644-8562-424253a015c3" containerName="kserve-container" containerID="cri-o://4d999282a0f644a5113ee038ee30cc3b5ec750cd64d9dc262e888c1b093ba902" gracePeriod=30
Jul 02 21:32:42.237738 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.237503    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" podUID="fa460ab6-5679-4644-8562-424253a015c3" containerName="kube-rbac-proxy" containerID="cri-o://66f3449808eff78bb56ccf828a053a6b293db121540f98b596d9f7185ee316d5" gracePeriod=30
Jul 02 21:32:42.239863 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.239843    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"]
Jul 02 21:32:42.248268 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.248148    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:32:42.275096 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.275067    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7273526a-482b-44bd-801d-dc5ce04f40f7-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd\" (UID: \"7273526a-482b-44bd-801d-dc5ce04f40f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:32:42.275337 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.275309    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7273526a-482b-44bd-801d-dc5ce04f40f7-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd\" (UID: \"7273526a-482b-44bd-801d-dc5ce04f40f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:32:42.275452 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.275388    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7273526a-482b-44bd-801d-dc5ce04f40f7-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd\" (UID: \"7273526a-482b-44bd-801d-dc5ce04f40f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:32:42.275452 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.275421    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8jn6\" (UniqueName: \"kubernetes.io/projected/7273526a-482b-44bd-801d-dc5ce04f40f7-kube-api-access-c8jn6\") pod \"isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd\" (UID: \"7273526a-482b-44bd-801d-dc5ce04f40f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:32:42.275561 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.275448    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7273526a-482b-44bd-801d-dc5ce04f40f7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd\" (UID: \"7273526a-482b-44bd-801d-dc5ce04f40f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:32:42.304644 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.304619    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"]
Jul 02 21:32:42.307920 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.307902    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:32:42.310582 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.310560    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-af6d2-predictor-serving-cert\""
Jul 02 21:32:42.310918 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.310903    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-af6d2-kube-rbac-proxy-sar-config\""
Jul 02 21:32:42.319975 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.319939    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"]
Jul 02 21:32:42.326450 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.326425    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:32:42.348199 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.348162    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"]
Jul 02 21:32:42.348442 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.348415    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw" podUID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerName="kserve-container" containerID="cri-o://7906be70aed2eb27bfa498a8db1d4966131d1ebe9bc707028696a68cdbb247df" gracePeriod=30
Jul 02 21:32:42.348553 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.348442    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw" podUID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerName="kube-rbac-proxy" containerID="cri-o://2520263b1b105194be4c9e96935caf5338551bdb90f49eded0c19b605688e54a" gracePeriod=30
Jul 02 21:32:42.376487 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.376458    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c\" (UID: \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:32:42.376577 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.376495    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c\" (UID: \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:32:42.376577 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.376521    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7273526a-482b-44bd-801d-dc5ce04f40f7-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd\" (UID: \"7273526a-482b-44bd-801d-dc5ce04f40f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:32:42.376577 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.376545    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k77gt\" (UniqueName: \"kubernetes.io/projected/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-kube-api-access-k77gt\") pod \"isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c\" (UID: \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:32:42.376680 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.376577    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7273526a-482b-44bd-801d-dc5ce04f40f7-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd\" (UID: \"7273526a-482b-44bd-801d-dc5ce04f40f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:32:42.376680 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.376648    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7273526a-482b-44bd-801d-dc5ce04f40f7-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd\" (UID: \"7273526a-482b-44bd-801d-dc5ce04f40f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:32:42.376755 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.376677    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c\" (UID: \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:32:42.376755 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.376711    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8jn6\" (UniqueName: \"kubernetes.io/projected/7273526a-482b-44bd-801d-dc5ce04f40f7-kube-api-access-c8jn6\") pod \"isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd\" (UID: \"7273526a-482b-44bd-801d-dc5ce04f40f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:32:42.376755 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.376743    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7273526a-482b-44bd-801d-dc5ce04f40f7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd\" (UID: \"7273526a-482b-44bd-801d-dc5ce04f40f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:32:42.376903 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.376771    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c\" (UID: \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:32:42.376903 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:32:42.376814    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-af6d2-predictor-serving-cert" not found
Jul 02 21:32:42.376903 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:32:42.376876    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7273526a-482b-44bd-801d-dc5ce04f40f7-proxy-tls podName:7273526a-482b-44bd-801d-dc5ce04f40f7 nodeName:}" failed. No retries permitted until 2026-07-02 21:32:42.876856388 +0000 UTC m=+834.503358243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7273526a-482b-44bd-801d-dc5ce04f40f7-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" (UID: "7273526a-482b-44bd-801d-dc5ce04f40f7") : secret "isvc-sklearn-graph-raw-hpa-af6d2-predictor-serving-cert" not found
Jul 02 21:32:42.376903 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.376880    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7273526a-482b-44bd-801d-dc5ce04f40f7-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd\" (UID: \"7273526a-482b-44bd-801d-dc5ce04f40f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:32:42.377279 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.377260    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7273526a-482b-44bd-801d-dc5ce04f40f7-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd\" (UID: \"7273526a-482b-44bd-801d-dc5ce04f40f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:32:42.377344 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.377327    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7273526a-482b-44bd-801d-dc5ce04f40f7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd\" (UID: \"7273526a-482b-44bd-801d-dc5ce04f40f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:32:42.386759 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.386738    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8jn6\" (UniqueName: \"kubernetes.io/projected/7273526a-482b-44bd-801d-dc5ce04f40f7-kube-api-access-c8jn6\") pod \"isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd\" (UID: \"7273526a-482b-44bd-801d-dc5ce04f40f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:32:42.393801 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.393779    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:32:42.431074 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.431044    2575 generic.go:356] "Generic (PLEG): container finished" podID="fa460ab6-5679-4644-8562-424253a015c3" containerID="66f3449808eff78bb56ccf828a053a6b293db121540f98b596d9f7185ee316d5" exitCode=2
Jul 02 21:32:42.431204 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.431109    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" event={"ID":"fa460ab6-5679-4644-8562-424253a015c3","Type":"ContainerDied","Data":"66f3449808eff78bb56ccf828a053a6b293db121540f98b596d9f7185ee316d5"}
Jul 02 21:32:42.477293 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.477265    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c\" (UID: \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:32:42.477416 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.477318    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c\" (UID: \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:32:42.477416 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.477384    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c\" (UID: \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:32:42.477556 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.477423    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c\" (UID: \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:32:42.477556 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.477456    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k77gt\" (UniqueName: \"kubernetes.io/projected/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-kube-api-access-k77gt\") pod \"isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c\" (UID: \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:32:42.477556 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:32:42.477475    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-af6d2-predictor-serving-cert" not found
Jul 02 21:32:42.477556 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:32:42.477555    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-proxy-tls podName:89e82f6f-f14e-4426-9c7d-9bc4fee4efb5 nodeName:}" failed. No retries permitted until 2026-07-02 21:32:42.977535629 +0000 UTC m=+834.604037484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c" (UID: "89e82f6f-f14e-4426-9c7d-9bc4fee4efb5") : secret "isvc-xgboost-graph-raw-hpa-af6d2-predictor-serving-cert" not found
Jul 02 21:32:42.478074 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.478005    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c\" (UID: \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:32:42.478203 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.478162    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c\" (UID: \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:32:42.478275 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.478201    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c\" (UID: \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:32:42.486065 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.486041    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k77gt\" (UniqueName: \"kubernetes.io/projected/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-kube-api-access-k77gt\") pod \"isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c\" (UID: \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:32:42.573796 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.573762    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:32:42.869115 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.869034    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:32:42.881221 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.881187    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7273526a-482b-44bd-801d-dc5ce04f40f7-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd\" (UID: \"7273526a-482b-44bd-801d-dc5ce04f40f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:32:42.883490 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.883470    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7273526a-482b-44bd-801d-dc5ce04f40f7-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd\" (UID: \"7273526a-482b-44bd-801d-dc5ce04f40f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:32:42.982401 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.982372    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c\" (UID: \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:32:42.984578 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.984554    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c\" (UID: \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:32:42.985607 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:42.985582    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" podUID="fa460ab6-5679-4644-8562-424253a015c3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 02 21:32:43.080383 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:43.080353    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw" podUID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 02 21:32:43.084726 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:43.084705    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw" podUID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 02 21:32:43.138809 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:43.138752    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:32:43.221480 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:43.221455    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:32:43.269159 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:43.269135    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"]
Jul 02 21:32:43.271913 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:32:43.271885    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7273526a_482b_44bd_801d_dc5ce04f40f7.slice/crio-67267b59fde28e105fe0ab0c65694620beee9185759d7830804ccfccf40ac57a WatchSource:0}: Error finding container 67267b59fde28e105fe0ab0c65694620beee9185759d7830804ccfccf40ac57a: Status 404 returned error can't find the container with id 67267b59fde28e105fe0ab0c65694620beee9185759d7830804ccfccf40ac57a
Jul 02 21:32:43.355236 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:43.355160    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"]
Jul 02 21:32:43.357072 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:32:43.357049    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e82f6f_f14e_4426_9c7d_9bc4fee4efb5.slice/crio-fc4cefaa106b60bdd5f8b8f28fc883975acdfa26abe1ecfd991c774369decc5c WatchSource:0}: Error finding container fc4cefaa106b60bdd5f8b8f28fc883975acdfa26abe1ecfd991c774369decc5c: Status 404 returned error can't find the container with id fc4cefaa106b60bdd5f8b8f28fc883975acdfa26abe1ecfd991c774369decc5c
Jul 02 21:32:43.437540 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:43.437513    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" event={"ID":"7273526a-482b-44bd-801d-dc5ce04f40f7","Type":"ContainerStarted","Data":"c0d4f02792d19dc7a8704584cbac88eed28f65ea4986c69a197f34aef151e3a1"}
Jul 02 21:32:43.437681 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:43.437578    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" event={"ID":"7273526a-482b-44bd-801d-dc5ce04f40f7","Type":"ContainerStarted","Data":"67267b59fde28e105fe0ab0c65694620beee9185759d7830804ccfccf40ac57a"}
Jul 02 21:32:43.439874 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:43.439835    2575 generic.go:356] "Generic (PLEG): container finished" podID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerID="2520263b1b105194be4c9e96935caf5338551bdb90f49eded0c19b605688e54a" exitCode=2
Jul 02 21:32:43.439974 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:43.439888    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw" event={"ID":"f252efa2-359f-4eb0-8a51-c2cf43f952af","Type":"ContainerDied","Data":"2520263b1b105194be4c9e96935caf5338551bdb90f49eded0c19b605688e54a"}
Jul 02 21:32:43.441757 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:43.441733    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c" event={"ID":"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5","Type":"ContainerStarted","Data":"4e8676ed1b0ecc9e531292e15aed8cad8470409883704d60edbad6a8463ffb1d"}
Jul 02 21:32:43.441865 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:43.441787    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c" event={"ID":"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5","Type":"ContainerStarted","Data":"fc4cefaa106b60bdd5f8b8f28fc883975acdfa26abe1ecfd991c774369decc5c"}
Jul 02 21:32:45.888458 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:45.888439    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:32:45.909699 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:45.909670    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f252efa2-359f-4eb0-8a51-c2cf43f952af-kube-rbac-proxy-sar-config\") pod \"f252efa2-359f-4eb0-8a51-c2cf43f952af\" (UID: \"f252efa2-359f-4eb0-8a51-c2cf43f952af\") "
Jul 02 21:32:45.909833 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:45.909715    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ht98\" (UniqueName: \"kubernetes.io/projected/f252efa2-359f-4eb0-8a51-c2cf43f952af-kube-api-access-2ht98\") pod \"f252efa2-359f-4eb0-8a51-c2cf43f952af\" (UID: \"f252efa2-359f-4eb0-8a51-c2cf43f952af\") "
Jul 02 21:32:45.909833 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:45.909750    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f252efa2-359f-4eb0-8a51-c2cf43f952af-kserve-provision-location\") pod \"f252efa2-359f-4eb0-8a51-c2cf43f952af\" (UID: \"f252efa2-359f-4eb0-8a51-c2cf43f952af\") "
Jul 02 21:32:45.909833 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:45.909783    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f252efa2-359f-4eb0-8a51-c2cf43f952af-cabundle-cert\") pod \"f252efa2-359f-4eb0-8a51-c2cf43f952af\" (UID: \"f252efa2-359f-4eb0-8a51-c2cf43f952af\") "
Jul 02 21:32:45.909833 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:45.909826    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f252efa2-359f-4eb0-8a51-c2cf43f952af-proxy-tls\") pod \"f252efa2-359f-4eb0-8a51-c2cf43f952af\" (UID: \"f252efa2-359f-4eb0-8a51-c2cf43f952af\") "
Jul 02 21:32:45.910048 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:45.909973    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f252efa2-359f-4eb0-8a51-c2cf43f952af-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f252efa2-359f-4eb0-8a51-c2cf43f952af" (UID: "f252efa2-359f-4eb0-8a51-c2cf43f952af"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:32:45.910112 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:45.910091    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f252efa2-359f-4eb0-8a51-c2cf43f952af-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f252efa2-359f-4eb0-8a51-c2cf43f952af" (UID: "f252efa2-359f-4eb0-8a51-c2cf43f952af"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:32:45.910194 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:45.910153    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f252efa2-359f-4eb0-8a51-c2cf43f952af-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f252efa2-359f-4eb0-8a51-c2cf43f952af" (UID: "f252efa2-359f-4eb0-8a51-c2cf43f952af"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:32:45.912228 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:45.912201    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f252efa2-359f-4eb0-8a51-c2cf43f952af-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f252efa2-359f-4eb0-8a51-c2cf43f952af" (UID: "f252efa2-359f-4eb0-8a51-c2cf43f952af"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:32:45.912380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:45.912263    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f252efa2-359f-4eb0-8a51-c2cf43f952af-kube-api-access-2ht98" (OuterVolumeSpecName: "kube-api-access-2ht98") pod "f252efa2-359f-4eb0-8a51-c2cf43f952af" (UID: "f252efa2-359f-4eb0-8a51-c2cf43f952af"). InnerVolumeSpecName "kube-api-access-2ht98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:32:46.009249 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.009217    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" podUID="fa460ab6-5679-4644-8562-424253a015c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 02 21:32:46.010561 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.010543    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f252efa2-359f-4eb0-8a51-c2cf43f952af-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:32:46.010634 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.010563    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ht98\" (UniqueName: \"kubernetes.io/projected/f252efa2-359f-4eb0-8a51-c2cf43f952af-kube-api-access-2ht98\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:32:46.010634 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.010573    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f252efa2-359f-4eb0-8a51-c2cf43f952af-kserve-provision-location\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:32:46.010634 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.010582    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f252efa2-359f-4eb0-8a51-c2cf43f952af-cabundle-cert\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:32:46.010634 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.010591    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f252efa2-359f-4eb0-8a51-c2cf43f952af-proxy-tls\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:32:46.457075 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.457045    2575 generic.go:356] "Generic (PLEG): container finished" podID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerID="7906be70aed2eb27bfa498a8db1d4966131d1ebe9bc707028696a68cdbb247df" exitCode=0
Jul 02 21:32:46.457215 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.457129    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"
Jul 02 21:32:46.457215 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.457126    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw" event={"ID":"f252efa2-359f-4eb0-8a51-c2cf43f952af","Type":"ContainerDied","Data":"7906be70aed2eb27bfa498a8db1d4966131d1ebe9bc707028696a68cdbb247df"}
Jul 02 21:32:46.457339 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.457264    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw" event={"ID":"f252efa2-359f-4eb0-8a51-c2cf43f952af","Type":"ContainerDied","Data":"ade2ac12a7651f52bfcd288c57dcad7d7c772095a0c039b9773b9e168e76f8e5"}
Jul 02 21:32:46.457339 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.457310    2575 scope.go:117] "RemoveContainer" containerID="2520263b1b105194be4c9e96935caf5338551bdb90f49eded0c19b605688e54a"
Jul 02 21:32:46.459728 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.459707    2575 generic.go:356] "Generic (PLEG): container finished" podID="fa460ab6-5679-4644-8562-424253a015c3" containerID="4d999282a0f644a5113ee038ee30cc3b5ec750cd64d9dc262e888c1b093ba902" exitCode=0
Jul 02 21:32:46.459845 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.459774    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" event={"ID":"fa460ab6-5679-4644-8562-424253a015c3","Type":"ContainerDied","Data":"4d999282a0f644a5113ee038ee30cc3b5ec750cd64d9dc262e888c1b093ba902"}
Jul 02 21:32:46.466238 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.466218    2575 scope.go:117] "RemoveContainer" containerID="7906be70aed2eb27bfa498a8db1d4966131d1ebe9bc707028696a68cdbb247df"
Jul 02 21:32:46.473668 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.473650    2575 scope.go:117] "RemoveContainer" containerID="d98b7918781cb34e0328620b80664d5822f9af13344766762f5d3e269cd50460"
Jul 02 21:32:46.481562 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.481535    2575 scope.go:117] "RemoveContainer" containerID="2520263b1b105194be4c9e96935caf5338551bdb90f49eded0c19b605688e54a"
Jul 02 21:32:46.481838 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:32:46.481818    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2520263b1b105194be4c9e96935caf5338551bdb90f49eded0c19b605688e54a\": container with ID starting with 2520263b1b105194be4c9e96935caf5338551bdb90f49eded0c19b605688e54a not found: ID does not exist" containerID="2520263b1b105194be4c9e96935caf5338551bdb90f49eded0c19b605688e54a"
Jul 02 21:32:46.481900 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.481847    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2520263b1b105194be4c9e96935caf5338551bdb90f49eded0c19b605688e54a"} err="failed to get container status \"2520263b1b105194be4c9e96935caf5338551bdb90f49eded0c19b605688e54a\": rpc error: code = NotFound desc = could not find container \"2520263b1b105194be4c9e96935caf5338551bdb90f49eded0c19b605688e54a\": container with ID starting with 2520263b1b105194be4c9e96935caf5338551bdb90f49eded0c19b605688e54a not found: ID does not exist"
Jul 02 21:32:46.481900 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.481863    2575 scope.go:117] "RemoveContainer" containerID="7906be70aed2eb27bfa498a8db1d4966131d1ebe9bc707028696a68cdbb247df"
Jul 02 21:32:46.482008 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.481831    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"]
Jul 02 21:32:46.482113 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:32:46.482097    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7906be70aed2eb27bfa498a8db1d4966131d1ebe9bc707028696a68cdbb247df\": container with ID starting with 7906be70aed2eb27bfa498a8db1d4966131d1ebe9bc707028696a68cdbb247df not found: ID does not exist" containerID="7906be70aed2eb27bfa498a8db1d4966131d1ebe9bc707028696a68cdbb247df"
Jul 02 21:32:46.482160 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.482118    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7906be70aed2eb27bfa498a8db1d4966131d1ebe9bc707028696a68cdbb247df"} err="failed to get container status \"7906be70aed2eb27bfa498a8db1d4966131d1ebe9bc707028696a68cdbb247df\": rpc error: code = NotFound desc = could not find container \"7906be70aed2eb27bfa498a8db1d4966131d1ebe9bc707028696a68cdbb247df\": container with ID starting with 7906be70aed2eb27bfa498a8db1d4966131d1ebe9bc707028696a68cdbb247df not found: ID does not exist"
Jul 02 21:32:46.482160 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.482130    2575 scope.go:117] "RemoveContainer" containerID="d98b7918781cb34e0328620b80664d5822f9af13344766762f5d3e269cd50460"
Jul 02 21:32:46.482381 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:32:46.482364    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d98b7918781cb34e0328620b80664d5822f9af13344766762f5d3e269cd50460\": container with ID starting with d98b7918781cb34e0328620b80664d5822f9af13344766762f5d3e269cd50460 not found: ID does not exist" containerID="d98b7918781cb34e0328620b80664d5822f9af13344766762f5d3e269cd50460"
Jul 02 21:32:46.482422 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.482384    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d98b7918781cb34e0328620b80664d5822f9af13344766762f5d3e269cd50460"} err="failed to get container status \"d98b7918781cb34e0328620b80664d5822f9af13344766762f5d3e269cd50460\": rpc error: code = NotFound desc = could not find container \"d98b7918781cb34e0328620b80664d5822f9af13344766762f5d3e269cd50460\": container with ID starting with d98b7918781cb34e0328620b80664d5822f9af13344766762f5d3e269cd50460 not found: ID does not exist"
Jul 02 21:32:46.485122 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.485103    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c1b1e-predictor-6976897857-2lbrw"]
Jul 02 21:32:46.573694 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.573663    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:32:46.615874 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.615820    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa460ab6-5679-4644-8562-424253a015c3-kube-rbac-proxy-sar-config\") pod \"fa460ab6-5679-4644-8562-424253a015c3\" (UID: \"fa460ab6-5679-4644-8562-424253a015c3\") "
Jul 02 21:32:46.615874 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.615859    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa460ab6-5679-4644-8562-424253a015c3-kserve-provision-location\") pod \"fa460ab6-5679-4644-8562-424253a015c3\" (UID: \"fa460ab6-5679-4644-8562-424253a015c3\") "
Jul 02 21:32:46.616030 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.615889    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa460ab6-5679-4644-8562-424253a015c3-cabundle-cert\") pod \"fa460ab6-5679-4644-8562-424253a015c3\" (UID: \"fa460ab6-5679-4644-8562-424253a015c3\") "
Jul 02 21:32:46.616030 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.615948    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa460ab6-5679-4644-8562-424253a015c3-proxy-tls\") pod \"fa460ab6-5679-4644-8562-424253a015c3\" (UID: \"fa460ab6-5679-4644-8562-424253a015c3\") "
Jul 02 21:32:46.616030 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.615975    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkfn6\" (UniqueName: \"kubernetes.io/projected/fa460ab6-5679-4644-8562-424253a015c3-kube-api-access-hkfn6\") pod \"fa460ab6-5679-4644-8562-424253a015c3\" (UID: \"fa460ab6-5679-4644-8562-424253a015c3\") "
Jul 02 21:32:46.616267 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.616242    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa460ab6-5679-4644-8562-424253a015c3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa460ab6-5679-4644-8562-424253a015c3" (UID: "fa460ab6-5679-4644-8562-424253a015c3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:32:46.616343 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.616271    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa460ab6-5679-4644-8562-424253a015c3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fa460ab6-5679-4644-8562-424253a015c3" (UID: "fa460ab6-5679-4644-8562-424253a015c3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:32:46.616343 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.616312    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa460ab6-5679-4644-8562-424253a015c3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fa460ab6-5679-4644-8562-424253a015c3" (UID: "fa460ab6-5679-4644-8562-424253a015c3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:32:46.617944 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.617925    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa460ab6-5679-4644-8562-424253a015c3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa460ab6-5679-4644-8562-424253a015c3" (UID: "fa460ab6-5679-4644-8562-424253a015c3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:32:46.618013 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.617954    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa460ab6-5679-4644-8562-424253a015c3-kube-api-access-hkfn6" (OuterVolumeSpecName: "kube-api-access-hkfn6") pod "fa460ab6-5679-4644-8562-424253a015c3" (UID: "fa460ab6-5679-4644-8562-424253a015c3"). InnerVolumeSpecName "kube-api-access-hkfn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:32:46.717284 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.717257    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa460ab6-5679-4644-8562-424253a015c3-proxy-tls\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:32:46.717284 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.717282    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkfn6\" (UniqueName: \"kubernetes.io/projected/fa460ab6-5679-4644-8562-424253a015c3-kube-api-access-hkfn6\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:32:46.717411 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.717298    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa460ab6-5679-4644-8562-424253a015c3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:32:46.717411 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.717311    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa460ab6-5679-4644-8562-424253a015c3-kserve-provision-location\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:32:46.717411 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:46.717324    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa460ab6-5679-4644-8562-424253a015c3-cabundle-cert\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:32:47.013989 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:47.013949    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f252efa2-359f-4eb0-8a51-c2cf43f952af" path="/var/lib/kubelet/pods/f252efa2-359f-4eb0-8a51-c2cf43f952af/volumes"
Jul 02 21:32:47.468598 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:47.468564    2575 generic.go:356] "Generic (PLEG): container finished" podID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerID="c0d4f02792d19dc7a8704584cbac88eed28f65ea4986c69a197f34aef151e3a1" exitCode=0
Jul 02 21:32:47.468789 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:47.468640    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" event={"ID":"7273526a-482b-44bd-801d-dc5ce04f40f7","Type":"ContainerDied","Data":"c0d4f02792d19dc7a8704584cbac88eed28f65ea4986c69a197f34aef151e3a1"}
Jul 02 21:32:47.471540 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:47.471520    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn" event={"ID":"fa460ab6-5679-4644-8562-424253a015c3","Type":"ContainerDied","Data":"95b66eb627e99ca841c4deb7dab8d44064505ceff181b304ec086baa3c0f3a8c"}
Jul 02 21:32:47.471631 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:47.471575    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"
Jul 02 21:32:47.471631 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:47.471590    2575 scope.go:117] "RemoveContainer" containerID="66f3449808eff78bb56ccf828a053a6b293db121540f98b596d9f7185ee316d5"
Jul 02 21:32:47.473092 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:47.473064    2575 generic.go:356] "Generic (PLEG): container finished" podID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" containerID="4e8676ed1b0ecc9e531292e15aed8cad8470409883704d60edbad6a8463ffb1d" exitCode=0
Jul 02 21:32:47.473243 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:47.473137    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c" event={"ID":"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5","Type":"ContainerDied","Data":"4e8676ed1b0ecc9e531292e15aed8cad8470409883704d60edbad6a8463ffb1d"}
Jul 02 21:32:47.482946 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:47.482930    2575 scope.go:117] "RemoveContainer" containerID="4d999282a0f644a5113ee038ee30cc3b5ec750cd64d9dc262e888c1b093ba902"
Jul 02 21:32:47.501979 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:47.501955    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"]
Jul 02 21:32:47.506311 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:47.506294    2575 scope.go:117] "RemoveContainer" containerID="7503c427c9369b3598085508b8869826196307e9981d2559a2c54443747417df"
Jul 02 21:32:47.507811 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:47.507793    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c1b1e-predictor-56f778f445-v8mhn"]
Jul 02 21:32:48.480374 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:48.480341    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c" event={"ID":"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5","Type":"ContainerStarted","Data":"3db017ece678767d8e558a9e1b88ddab9bede14a71dc19ac4df343bf40f32da2"}
Jul 02 21:32:48.480793 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:48.480404    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c" event={"ID":"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5","Type":"ContainerStarted","Data":"4746413e86cd6efa6a1bb6094daf9ef4d8d6e601e6e00129a96ef6208ed53190"}
Jul 02 21:32:48.480793 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:48.480608    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:32:48.482398 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:48.482375    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" event={"ID":"7273526a-482b-44bd-801d-dc5ce04f40f7","Type":"ContainerStarted","Data":"ce97fadf7c3a6b0ae7478ee802a38909c05f656786bd76d2879da4677d41f2ba"}
Jul 02 21:32:48.482542 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:48.482430    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" event={"ID":"7273526a-482b-44bd-801d-dc5ce04f40f7","Type":"ContainerStarted","Data":"3d3272c10e66cbaaa91431bcdc927f22c8ce36843b74f29398fdb255a6788987"}
Jul 02 21:32:48.482667 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:48.482643    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:32:48.482781 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:48.482692    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:32:48.483991 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:48.483967    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 21:32:48.511143 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:48.511097    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c" podStartSLOduration=2.630201236 podStartE2EDuration="6.51108587s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.880884634s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:32:48.51048537 +0000 UTC m=+840.136987241" watchObservedRunningTime="2026-07-02 21:32:48.51108587 +0000 UTC m=+840.137587746"
Jul 02 21:32:48.531426 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:48.531378    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" podStartSLOduration=2.81569521 podStartE2EDuration="6.531364315s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.715669105s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:32:48.529561267 +0000 UTC m=+840.156063193" watchObservedRunningTime="2026-07-02 21:32:48.531364315 +0000 UTC m=+840.157866190"
Jul 02 21:32:49.014032 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:49.014004    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa460ab6-5679-4644-8562-424253a015c3" path="/var/lib/kubelet/pods/fa460ab6-5679-4644-8562-424253a015c3/volumes"
Jul 02 21:32:49.487264 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:49.487217    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 21:32:49.487675 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:49.487356    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:32:49.488645 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:49.488624    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c" podUID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 02 21:32:50.491030 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:50.490986    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c" podUID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 02 21:32:54.491160 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:54.491130    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:32:54.491683 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:54.491659    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 21:32:55.495668 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:55.495639    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:32:55.496131 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:32:55.496104    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c" podUID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 02 21:33:04.491901 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:33:04.491866    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 21:33:05.497065 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:33:05.497027    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c" podUID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 02 21:33:14.491644 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:33:14.491607    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 21:33:15.497004 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:33:15.496955    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c" podUID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 02 21:33:24.491926 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:33:24.491882    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 21:33:25.496703 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:33:25.496668    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c" podUID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 02 21:33:34.492413 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:33:34.492371    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 21:33:35.496988 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:33:35.496942    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c" podUID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 02 21:33:44.492549 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:33:44.492469    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 21:33:45.497324 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:33:45.497293    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:33:54.492314 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:33:54.492270    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 21:33:56.009963 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:33:56.009919    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 21:34:06.010958 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:06.010929    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:34:22.246604 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:22.246565    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:34:22.419052 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:22.419023    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:34:32.286891 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.286856    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:34:32.474210 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.474178    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"]
Jul 02 21:34:32.474533 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.474520    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa460ab6-5679-4644-8562-424253a015c3" containerName="kube-rbac-proxy"
Jul 02 21:34:32.474588 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.474534    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa460ab6-5679-4644-8562-424253a015c3" containerName="kube-rbac-proxy"
Jul 02 21:34:32.474588 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.474556    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa460ab6-5679-4644-8562-424253a015c3" containerName="kserve-container"
Jul 02 21:34:32.474588 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.474565    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa460ab6-5679-4644-8562-424253a015c3" containerName="kserve-container"
Jul 02 21:34:32.474725 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.474593    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa460ab6-5679-4644-8562-424253a015c3" containerName="storage-initializer"
Jul 02 21:34:32.474725 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.474602    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa460ab6-5679-4644-8562-424253a015c3" containerName="storage-initializer"
Jul 02 21:34:32.474725 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.474612    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerName="kserve-container"
Jul 02 21:34:32.474725 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.474620    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerName="kserve-container"
Jul 02 21:34:32.474725 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.474636    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerName="kube-rbac-proxy"
Jul 02 21:34:32.474725 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.474642    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerName="kube-rbac-proxy"
Jul 02 21:34:32.474725 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.474651    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerName="storage-initializer"
Jul 02 21:34:32.474725 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.474657    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerName="storage-initializer"
Jul 02 21:34:32.474725 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.474725    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa460ab6-5679-4644-8562-424253a015c3" containerName="kube-rbac-proxy"
Jul 02 21:34:32.475001 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.474735    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerName="kserve-container"
Jul 02 21:34:32.475001 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.474747    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa460ab6-5679-4644-8562-424253a015c3" containerName="kserve-container"
Jul 02 21:34:32.475001 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.474755    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f252efa2-359f-4eb0-8a51-c2cf43f952af" containerName="kube-rbac-proxy"
Jul 02 21:34:32.477551 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.477532    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"
Jul 02 21:34:32.481024 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.481003    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-1f359-predictor-serving-cert\""
Jul 02 21:34:32.481572 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.481551    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-1f359-kube-rbac-proxy-sar-config\""
Jul 02 21:34:32.489411 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.489389    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"]
Jul 02 21:34:32.553315 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.553253    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"]
Jul 02 21:34:32.553553 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.553509    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerName="kserve-container" containerID="cri-o://3d3272c10e66cbaaa91431bcdc927f22c8ce36843b74f29398fdb255a6788987" gracePeriod=30
Jul 02 21:34:32.553617 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.553593    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerName="kube-rbac-proxy" containerID="cri-o://ce97fadf7c3a6b0ae7478ee802a38909c05f656786bd76d2879da4677d41f2ba" gracePeriod=30
Jul 02 21:34:32.609117 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.609087    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"]
Jul 02 21:34:32.609396 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.609366    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c" podUID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" containerName="kserve-container" containerID="cri-o://4746413e86cd6efa6a1bb6094daf9ef4d8d6e601e6e00129a96ef6208ed53190" gracePeriod=30
Jul 02 21:34:32.609510 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.609413    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c" podUID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" containerName="kube-rbac-proxy" containerID="cri-o://3db017ece678767d8e558a9e1b88ddab9bede14a71dc19ac4df343bf40f32da2" gracePeriod=30
Jul 02 21:34:32.652927 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.652894    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceb934c5-0868-411c-bb37-3b3f3f2bf665-proxy-tls\") pod \"message-dumper-raw-1f359-predictor-864d8c9886-4cb52\" (UID: \"ceb934c5-0868-411c-bb37-3b3f3f2bf665\") " pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"
Jul 02 21:34:32.653052 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.652955    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ceb934c5-0868-411c-bb37-3b3f3f2bf665-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-1f359-predictor-864d8c9886-4cb52\" (UID: \"ceb934c5-0868-411c-bb37-3b3f3f2bf665\") " pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"
Jul 02 21:34:32.653052 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.652991    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbsnb\" (UniqueName: \"kubernetes.io/projected/ceb934c5-0868-411c-bb37-3b3f3f2bf665-kube-api-access-mbsnb\") pod \"message-dumper-raw-1f359-predictor-864d8c9886-4cb52\" (UID: \"ceb934c5-0868-411c-bb37-3b3f3f2bf665\") " pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"
Jul 02 21:34:32.754101 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.754064    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ceb934c5-0868-411c-bb37-3b3f3f2bf665-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-1f359-predictor-864d8c9886-4cb52\" (UID: \"ceb934c5-0868-411c-bb37-3b3f3f2bf665\") " pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"
Jul 02 21:34:32.754250 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.754121    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbsnb\" (UniqueName: \"kubernetes.io/projected/ceb934c5-0868-411c-bb37-3b3f3f2bf665-kube-api-access-mbsnb\") pod \"message-dumper-raw-1f359-predictor-864d8c9886-4cb52\" (UID: \"ceb934c5-0868-411c-bb37-3b3f3f2bf665\") " pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"
Jul 02 21:34:32.754321 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.754250    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceb934c5-0868-411c-bb37-3b3f3f2bf665-proxy-tls\") pod \"message-dumper-raw-1f359-predictor-864d8c9886-4cb52\" (UID: \"ceb934c5-0868-411c-bb37-3b3f3f2bf665\") " pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"
Jul 02 21:34:32.754406 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:34:32.754389    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-serving-cert: secret "message-dumper-raw-1f359-predictor-serving-cert" not found
Jul 02 21:34:32.754473 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:34:32.754459    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceb934c5-0868-411c-bb37-3b3f3f2bf665-proxy-tls podName:ceb934c5-0868-411c-bb37-3b3f3f2bf665 nodeName:}" failed. No retries permitted until 2026-07-02 21:34:33.25443991 +0000 UTC m=+944.880941782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ceb934c5-0868-411c-bb37-3b3f3f2bf665-proxy-tls") pod "message-dumper-raw-1f359-predictor-864d8c9886-4cb52" (UID: "ceb934c5-0868-411c-bb37-3b3f3f2bf665") : secret "message-dumper-raw-1f359-predictor-serving-cert" not found
Jul 02 21:34:32.754865 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.754843    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ceb934c5-0868-411c-bb37-3b3f3f2bf665-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-1f359-predictor-864d8c9886-4cb52\" (UID: \"ceb934c5-0868-411c-bb37-3b3f3f2bf665\") " pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"
Jul 02 21:34:32.763393 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.763362    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbsnb\" (UniqueName: \"kubernetes.io/projected/ceb934c5-0868-411c-bb37-3b3f3f2bf665-kube-api-access-mbsnb\") pod \"message-dumper-raw-1f359-predictor-864d8c9886-4cb52\" (UID: \"ceb934c5-0868-411c-bb37-3b3f3f2bf665\") " pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"
Jul 02 21:34:32.911016 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.910987    2575 generic.go:356] "Generic (PLEG): container finished" podID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" containerID="3db017ece678767d8e558a9e1b88ddab9bede14a71dc19ac4df343bf40f32da2" exitCode=2
Jul 02 21:34:32.911163 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.911059    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c" event={"ID":"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5","Type":"ContainerDied","Data":"3db017ece678767d8e558a9e1b88ddab9bede14a71dc19ac4df343bf40f32da2"}
Jul 02 21:34:32.913116 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.913089    2575 generic.go:356] "Generic (PLEG): container finished" podID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerID="ce97fadf7c3a6b0ae7478ee802a38909c05f656786bd76d2879da4677d41f2ba" exitCode=2
Jul 02 21:34:32.913245 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:32.913144    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" event={"ID":"7273526a-482b-44bd-801d-dc5ce04f40f7","Type":"ContainerDied","Data":"ce97fadf7c3a6b0ae7478ee802a38909c05f656786bd76d2879da4677d41f2ba"}
Jul 02 21:34:33.259200 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:33.259109    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceb934c5-0868-411c-bb37-3b3f3f2bf665-proxy-tls\") pod \"message-dumper-raw-1f359-predictor-864d8c9886-4cb52\" (UID: \"ceb934c5-0868-411c-bb37-3b3f3f2bf665\") " pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"
Jul 02 21:34:33.261312 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:33.261284    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceb934c5-0868-411c-bb37-3b3f3f2bf665-proxy-tls\") pod \"message-dumper-raw-1f359-predictor-864d8c9886-4cb52\" (UID: \"ceb934c5-0868-411c-bb37-3b3f3f2bf665\") " pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"
Jul 02 21:34:33.387837 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:33.387811    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"
Jul 02 21:34:33.415074 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:33.415042    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:34:33.517890 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:33.517870    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"]
Jul 02 21:34:33.520204 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:34:33.520179    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb934c5_0868_411c_bb37_3b3f3f2bf665.slice/crio-1cc2a0c249c7c8f8101214de1e2f8b5ea8b0abeced64f77e10ddfa0627378e1c WatchSource:0}: Error finding container 1cc2a0c249c7c8f8101214de1e2f8b5ea8b0abeced64f77e10ddfa0627378e1c: Status 404 returned error can't find the container with id 1cc2a0c249c7c8f8101214de1e2f8b5ea8b0abeced64f77e10ddfa0627378e1c
Jul 02 21:34:33.522122 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:33.522108    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 21:34:33.918856 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:33.918823    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52" event={"ID":"ceb934c5-0868-411c-bb37-3b3f3f2bf665","Type":"ContainerStarted","Data":"1cc2a0c249c7c8f8101214de1e2f8b5ea8b0abeced64f77e10ddfa0627378e1c"}
Jul 02 21:34:34.487482 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:34.487436    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 02 21:34:35.491742 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:35.491695    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c" podUID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Jul 02 21:34:35.496033 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:35.496001    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c" podUID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 02 21:34:35.932028 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:35.931993    2575 generic.go:356] "Generic (PLEG): container finished" podID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" containerID="4746413e86cd6efa6a1bb6094daf9ef4d8d6e601e6e00129a96ef6208ed53190" exitCode=0
Jul 02 21:34:35.932151 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:35.932047    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c" event={"ID":"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5","Type":"ContainerDied","Data":"4746413e86cd6efa6a1bb6094daf9ef4d8d6e601e6e00129a96ef6208ed53190"}
Jul 02 21:34:35.933929 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:35.933909    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52" event={"ID":"ceb934c5-0868-411c-bb37-3b3f3f2bf665","Type":"ContainerStarted","Data":"c3d483ef0f9b8f7ad0442d072319fc4b08bf71e204160a790457b324bfcfc917"}
Jul 02 21:34:35.934034 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:35.933957    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52" event={"ID":"ceb934c5-0868-411c-bb37-3b3f3f2bf665","Type":"ContainerStarted","Data":"72f1b7dbe99c9e06f6e15fda439af48d626cbee16518a158801811c09cc972fd"}
Jul 02 21:34:35.934034 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:35.934021    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"
Jul 02 21:34:35.954006 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:35.953966    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52" podStartSLOduration=2.615761745 podStartE2EDuration="3.953951215s" totalImagesPullingTime="1.33818947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:34:35.953207768 +0000 UTC m=+947.579709644" watchObservedRunningTime="2026-07-02 21:34:35.953951215 +0000 UTC m=+947.580453091"
Jul 02 21:34:36.009924 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.009895    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 21:34:36.053786 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.053762    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:34:36.184757 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.184733    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-kube-rbac-proxy-sar-config\") pod \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\" (UID: \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\") "
Jul 02 21:34:36.184885 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.184766    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-kserve-provision-location\") pod \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\" (UID: \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\") "
Jul 02 21:34:36.184885 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.184822    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-cabundle-cert\") pod \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\" (UID: \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\") "
Jul 02 21:34:36.184885 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.184850    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k77gt\" (UniqueName: \"kubernetes.io/projected/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-kube-api-access-k77gt\") pod \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\" (UID: \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\") "
Jul 02 21:34:36.185013 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.184899    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-proxy-tls\") pod \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\" (UID: \"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5\") "
Jul 02 21:34:36.185145 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.185122    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" (UID: "89e82f6f-f14e-4426-9c7d-9bc4fee4efb5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:34:36.185242 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.185148    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" (UID: "89e82f6f-f14e-4426-9c7d-9bc4fee4efb5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:34:36.185301 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.185236    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" (UID: "89e82f6f-f14e-4426-9c7d-9bc4fee4efb5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:34:36.186979 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.186959    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-kube-api-access-k77gt" (OuterVolumeSpecName: "kube-api-access-k77gt") pod "89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" (UID: "89e82f6f-f14e-4426-9c7d-9bc4fee4efb5"). InnerVolumeSpecName "kube-api-access-k77gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:34:36.187077 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.187058    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" (UID: "89e82f6f-f14e-4426-9c7d-9bc4fee4efb5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:34:36.286411 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.286380    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k77gt\" (UniqueName: \"kubernetes.io/projected/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-kube-api-access-k77gt\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:34:36.286411 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.286405    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-proxy-tls\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:34:36.286411 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.286416    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:34:36.286635 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.286425    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-kserve-provision-location\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:34:36.286635 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.286434    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5-cabundle-cert\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:34:36.940816 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.940787    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"
Jul 02 21:34:36.941266 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.940807    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c" event={"ID":"89e82f6f-f14e-4426-9c7d-9bc4fee4efb5","Type":"ContainerDied","Data":"fc4cefaa106b60bdd5f8b8f28fc883975acdfa26abe1ecfd991c774369decc5c"}
Jul 02 21:34:36.941266 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.940903    2575 scope.go:117] "RemoveContainer" containerID="3db017ece678767d8e558a9e1b88ddab9bede14a71dc19ac4df343bf40f32da2"
Jul 02 21:34:36.943397 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.943371    2575 generic.go:356] "Generic (PLEG): container finished" podID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerID="3d3272c10e66cbaaa91431bcdc927f22c8ce36843b74f29398fdb255a6788987" exitCode=0
Jul 02 21:34:36.943519 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.943446    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" event={"ID":"7273526a-482b-44bd-801d-dc5ce04f40f7","Type":"ContainerDied","Data":"3d3272c10e66cbaaa91431bcdc927f22c8ce36843b74f29398fdb255a6788987"}
Jul 02 21:34:36.943684 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.943657    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"
Jul 02 21:34:36.945497 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.945482    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"
Jul 02 21:34:36.953134 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.952634    2575 scope.go:117] "RemoveContainer" containerID="4746413e86cd6efa6a1bb6094daf9ef4d8d6e601e6e00129a96ef6208ed53190"
Jul 02 21:34:36.961797 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.961778    2575 scope.go:117] "RemoveContainer" containerID="4e8676ed1b0ecc9e531292e15aed8cad8470409883704d60edbad6a8463ffb1d"
Jul 02 21:34:36.987479 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.987392    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"]
Jul 02 21:34:36.993200 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.993180    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-af6d2-predictor-69bcd97d57-59x9c"]
Jul 02 21:34:36.998876 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:36.998859    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:34:37.014135 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.014115    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" path="/var/lib/kubelet/pods/89e82f6f-f14e-4426-9c7d-9bc4fee4efb5/volumes"
Jul 02 21:34:37.092686 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.092637    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8jn6\" (UniqueName: \"kubernetes.io/projected/7273526a-482b-44bd-801d-dc5ce04f40f7-kube-api-access-c8jn6\") pod \"7273526a-482b-44bd-801d-dc5ce04f40f7\" (UID: \"7273526a-482b-44bd-801d-dc5ce04f40f7\") "
Jul 02 21:34:37.092775 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.092690    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7273526a-482b-44bd-801d-dc5ce04f40f7-kserve-provision-location\") pod \"7273526a-482b-44bd-801d-dc5ce04f40f7\" (UID: \"7273526a-482b-44bd-801d-dc5ce04f40f7\") "
Jul 02 21:34:37.092775 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.092717    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7273526a-482b-44bd-801d-dc5ce04f40f7-cabundle-cert\") pod \"7273526a-482b-44bd-801d-dc5ce04f40f7\" (UID: \"7273526a-482b-44bd-801d-dc5ce04f40f7\") "
Jul 02 21:34:37.092775 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.092741    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7273526a-482b-44bd-801d-dc5ce04f40f7-kube-rbac-proxy-sar-config\") pod \"7273526a-482b-44bd-801d-dc5ce04f40f7\" (UID: \"7273526a-482b-44bd-801d-dc5ce04f40f7\") "
Jul 02 21:34:37.092902 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.092882    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7273526a-482b-44bd-801d-dc5ce04f40f7-proxy-tls\") pod \"7273526a-482b-44bd-801d-dc5ce04f40f7\" (UID: \"7273526a-482b-44bd-801d-dc5ce04f40f7\") "
Jul 02 21:34:37.093047 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.093024    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7273526a-482b-44bd-801d-dc5ce04f40f7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7273526a-482b-44bd-801d-dc5ce04f40f7" (UID: "7273526a-482b-44bd-801d-dc5ce04f40f7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:34:37.093122 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.093031    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7273526a-482b-44bd-801d-dc5ce04f40f7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7273526a-482b-44bd-801d-dc5ce04f40f7" (UID: "7273526a-482b-44bd-801d-dc5ce04f40f7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:34:37.093122 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.093047    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7273526a-482b-44bd-801d-dc5ce04f40f7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7273526a-482b-44bd-801d-dc5ce04f40f7" (UID: "7273526a-482b-44bd-801d-dc5ce04f40f7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:34:37.093285 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.093252    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7273526a-482b-44bd-801d-dc5ce04f40f7-kserve-provision-location\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:34:37.093285 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.093278    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7273526a-482b-44bd-801d-dc5ce04f40f7-cabundle-cert\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:34:37.093448 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.093292    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7273526a-482b-44bd-801d-dc5ce04f40f7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:34:37.095246 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.095224    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7273526a-482b-44bd-801d-dc5ce04f40f7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7273526a-482b-44bd-801d-dc5ce04f40f7" (UID: "7273526a-482b-44bd-801d-dc5ce04f40f7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:34:37.095320 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.095248    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7273526a-482b-44bd-801d-dc5ce04f40f7-kube-api-access-c8jn6" (OuterVolumeSpecName: "kube-api-access-c8jn6") pod "7273526a-482b-44bd-801d-dc5ce04f40f7" (UID: "7273526a-482b-44bd-801d-dc5ce04f40f7"). InnerVolumeSpecName "kube-api-access-c8jn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:34:37.194448 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.194426    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7273526a-482b-44bd-801d-dc5ce04f40f7-proxy-tls\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:34:37.194448 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.194447    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8jn6\" (UniqueName: \"kubernetes.io/projected/7273526a-482b-44bd-801d-dc5ce04f40f7-kube-api-access-c8jn6\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:34:37.949832 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.949804    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"
Jul 02 21:34:37.950277 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.949827    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd" event={"ID":"7273526a-482b-44bd-801d-dc5ce04f40f7","Type":"ContainerDied","Data":"67267b59fde28e105fe0ab0c65694620beee9185759d7830804ccfccf40ac57a"}
Jul 02 21:34:37.950277 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.949894    2575 scope.go:117] "RemoveContainer" containerID="ce97fadf7c3a6b0ae7478ee802a38909c05f656786bd76d2879da4677d41f2ba"
Jul 02 21:34:37.958253 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.958229    2575 scope.go:117] "RemoveContainer" containerID="3d3272c10e66cbaaa91431bcdc927f22c8ce36843b74f29398fdb255a6788987"
Jul 02 21:34:37.965385 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.965370    2575 scope.go:117] "RemoveContainer" containerID="c0d4f02792d19dc7a8704584cbac88eed28f65ea4986c69a197f34aef151e3a1"
Jul 02 21:34:37.971934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.971912    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"]
Jul 02 21:34:37.975480 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:37.975457    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-af6d2-predictor-7bd4dcc697-mbgrd"]
Jul 02 21:34:39.014409 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:39.014378    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" path="/var/lib/kubelet/pods/7273526a-482b-44bd-801d-dc5ce04f40f7/volumes"
Jul 02 21:34:43.959401 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:43.959367    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"
Jul 02 21:34:52.456194 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.456145    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:34:52.505936 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.505906    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"]
Jul 02 21:34:52.506274 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.506253    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerName="kube-rbac-proxy"
Jul 02 21:34:52.506274 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.506269    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerName="kube-rbac-proxy"
Jul 02 21:34:52.506434 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.506282    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" containerName="kube-rbac-proxy"
Jul 02 21:34:52.506434 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.506289    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" containerName="kube-rbac-proxy"
Jul 02 21:34:52.506434 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.506298    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerName="kserve-container"
Jul 02 21:34:52.506434 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.506304    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerName="kserve-container"
Jul 02 21:34:52.506434 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.506312    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" containerName="storage-initializer"
Jul 02 21:34:52.506434 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.506317    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" containerName="storage-initializer"
Jul 02 21:34:52.506434 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.506325    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerName="storage-initializer"
Jul 02 21:34:52.506434 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.506330    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerName="storage-initializer"
Jul 02 21:34:52.506434 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.506340    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" containerName="kserve-container"
Jul 02 21:34:52.506434 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.506345    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" containerName="kserve-container"
Jul 02 21:34:52.506434 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.506416    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerName="kserve-container"
Jul 02 21:34:52.506434 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.506424    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" containerName="kube-rbac-proxy"
Jul 02 21:34:52.506434 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.506434    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="89e82f6f-f14e-4426-9c7d-9bc4fee4efb5" containerName="kserve-container"
Jul 02 21:34:52.506434 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.506440    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7273526a-482b-44bd-801d-dc5ce04f40f7" containerName="kube-rbac-proxy"
Jul 02 21:34:52.510786 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.510769    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:34:52.513919 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.513901    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-1f359-kube-rbac-proxy-sar-config\""
Jul 02 21:34:52.514240 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.514223    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 21:34:52.514354 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.514223    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 21:34:52.514681 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.514664    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-1f359-predictor-serving-cert\""
Jul 02 21:34:52.522150 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.522128    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"]
Jul 02 21:34:52.547499 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.547475    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:34:52.623624 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.623601    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3aadcb6a-96b7-456e-8eb7-2b710f299456-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6\" (UID: \"3aadcb6a-96b7-456e-8eb7-2b710f299456\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:34:52.623715 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.623630    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3aadcb6a-96b7-456e-8eb7-2b710f299456-kserve-provision-location\") pod \"isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6\" (UID: \"3aadcb6a-96b7-456e-8eb7-2b710f299456\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:34:52.623715 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.623663    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5thtp\" (UniqueName: \"kubernetes.io/projected/3aadcb6a-96b7-456e-8eb7-2b710f299456-kube-api-access-5thtp\") pod \"isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6\" (UID: \"3aadcb6a-96b7-456e-8eb7-2b710f299456\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:34:52.623806 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.623744    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3aadcb6a-96b7-456e-8eb7-2b710f299456-cabundle-cert\") pod \"isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6\" (UID: \"3aadcb6a-96b7-456e-8eb7-2b710f299456\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:34:52.623806 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.623777    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3aadcb6a-96b7-456e-8eb7-2b710f299456-proxy-tls\") pod \"isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6\" (UID: \"3aadcb6a-96b7-456e-8eb7-2b710f299456\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:34:52.725040 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.724989    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3aadcb6a-96b7-456e-8eb7-2b710f299456-proxy-tls\") pod \"isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6\" (UID: \"3aadcb6a-96b7-456e-8eb7-2b710f299456\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:34:52.725040 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.725034    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3aadcb6a-96b7-456e-8eb7-2b710f299456-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6\" (UID: \"3aadcb6a-96b7-456e-8eb7-2b710f299456\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:34:52.725145 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.725053    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3aadcb6a-96b7-456e-8eb7-2b710f299456-kserve-provision-location\") pod \"isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6\" (UID: \"3aadcb6a-96b7-456e-8eb7-2b710f299456\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:34:52.725220 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.725165    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5thtp\" (UniqueName: \"kubernetes.io/projected/3aadcb6a-96b7-456e-8eb7-2b710f299456-kube-api-access-5thtp\") pod \"isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6\" (UID: \"3aadcb6a-96b7-456e-8eb7-2b710f299456\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:34:52.725277 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.725239    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3aadcb6a-96b7-456e-8eb7-2b710f299456-cabundle-cert\") pod \"isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6\" (UID: \"3aadcb6a-96b7-456e-8eb7-2b710f299456\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:34:52.725436 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.725415    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3aadcb6a-96b7-456e-8eb7-2b710f299456-kserve-provision-location\") pod \"isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6\" (UID: \"3aadcb6a-96b7-456e-8eb7-2b710f299456\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:34:52.725738 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.725716    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3aadcb6a-96b7-456e-8eb7-2b710f299456-cabundle-cert\") pod \"isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6\" (UID: \"3aadcb6a-96b7-456e-8eb7-2b710f299456\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:34:52.725826 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.725747    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3aadcb6a-96b7-456e-8eb7-2b710f299456-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6\" (UID: \"3aadcb6a-96b7-456e-8eb7-2b710f299456\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:34:52.727492 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.727470    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3aadcb6a-96b7-456e-8eb7-2b710f299456-proxy-tls\") pod \"isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6\" (UID: \"3aadcb6a-96b7-456e-8eb7-2b710f299456\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:34:52.733515 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.733497    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5thtp\" (UniqueName: \"kubernetes.io/projected/3aadcb6a-96b7-456e-8eb7-2b710f299456-kube-api-access-5thtp\") pod \"isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6\" (UID: \"3aadcb6a-96b7-456e-8eb7-2b710f299456\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:34:52.820293 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.820272    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:34:52.949303 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:52.949231    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"]
Jul 02 21:34:52.951413 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:34:52.951388    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aadcb6a_96b7_456e_8eb7_2b710f299456.slice/crio-cf5a60137314c14378d74151210b4134980869a2b118b4b9e67aeca5b84c39e1 WatchSource:0}: Error finding container cf5a60137314c14378d74151210b4134980869a2b118b4b9e67aeca5b84c39e1: Status 404 returned error can't find the container with id cf5a60137314c14378d74151210b4134980869a2b118b4b9e67aeca5b84c39e1
Jul 02 21:34:53.014174 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:53.014145    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" event={"ID":"3aadcb6a-96b7-456e-8eb7-2b710f299456","Type":"ContainerStarted","Data":"d3d9dfc42d32cc49db285a11fa274c803418a007cfc7a7a9e86aede683a26fac"}
Jul 02 21:34:53.014272 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:53.014229    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" event={"ID":"3aadcb6a-96b7-456e-8eb7-2b710f299456","Type":"ContainerStarted","Data":"cf5a60137314c14378d74151210b4134980869a2b118b4b9e67aeca5b84c39e1"}
Jul 02 21:34:57.033675 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:57.033651    2575 generic.go:356] "Generic (PLEG): container finished" podID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerID="d3d9dfc42d32cc49db285a11fa274c803418a007cfc7a7a9e86aede683a26fac" exitCode=0
Jul 02 21:34:57.033994 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:57.033687    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" event={"ID":"3aadcb6a-96b7-456e-8eb7-2b710f299456","Type":"ContainerDied","Data":"d3d9dfc42d32cc49db285a11fa274c803418a007cfc7a7a9e86aede683a26fac"}
Jul 02 21:34:58.040716 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:58.040684    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" event={"ID":"3aadcb6a-96b7-456e-8eb7-2b710f299456","Type":"ContainerStarted","Data":"ef4d8b081cbfabd567d2c4d8b30b22c7dfd68e8a48642c942e9133b5582c93ed"}
Jul 02 21:34:58.041087 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:58.040748    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" event={"ID":"3aadcb6a-96b7-456e-8eb7-2b710f299456","Type":"ContainerStarted","Data":"421ee2ed43358d7ba1568daf7c34b920ad40f23df33564f3ba6d55e3dbd18780"}
Jul 02 21:34:58.041087 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:58.040772    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" event={"ID":"3aadcb6a-96b7-456e-8eb7-2b710f299456","Type":"ContainerStarted","Data":"a2049dfe85af2959d25aaa96f7b82ff9e6f7078a19da9e851eb1887542a443d9"}
Jul 02 21:34:58.041185 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:58.041096    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:34:58.041246 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:58.041226    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:34:58.042503 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:58.042479    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 21:34:58.062304 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:58.062247    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podStartSLOduration=2.265741195 podStartE2EDuration="6.062225172s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.796483977s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:34:58.061051182 +0000 UTC m=+969.687553090" watchObservedRunningTime="2026-07-02 21:34:58.062225172 +0000 UTC m=+969.688727049"
Jul 02 21:34:59.044196 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:59.044144    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 21:34:59.044562 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:59.044204    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:34:59.045397 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:34:59.045375    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:35:00.048522 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:35:00.048485    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 21:35:00.048947 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:35:00.048869    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:35:05.052680 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:35:05.052650    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:35:05.053339 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:35:05.053305    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 21:35:05.053511 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:35:05.053488    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:35:15.053608 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:35:15.053523    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 21:35:15.054061 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:35:15.053943    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:35:25.053941 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:35:25.053897    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 21:35:25.054442 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:35:25.054402    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:35:35.054038 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:35:35.054001    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 21:35:35.054526 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:35:35.054501    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:35:45.053960 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:35:45.053920    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 21:35:45.054441 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:35:45.054417    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:35:55.053381 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:35:55.053343    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 21:35:55.053845 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:35:55.053787    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:36:05.053589 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:05.053522    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 21:36:05.053998 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:05.053974    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:36:13.013742 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:13.013716    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:36:13.014147 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:13.013790    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:36:27.523985 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.523955    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-1f359-predictor-864d8c9886-4cb52_ceb934c5-0868-411c-bb37-3b3f3f2bf665/kserve-container/0.log"
Jul 02 21:36:27.762390 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.762357    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:36:27.807492 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.807412    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"]
Jul 02 21:36:27.810917 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.810897    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:36:27.813743 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.813721    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-e3b87-predictor-serving-cert\""
Jul 02 21:36:27.813879 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.813762    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-e3b87-kube-rbac-proxy-sar-config\""
Jul 02 21:36:27.819978 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.819957    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"]
Jul 02 21:36:27.820431 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.820387    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kserve-container" containerID="cri-o://a2049dfe85af2959d25aaa96f7b82ff9e6f7078a19da9e851eb1887542a443d9" gracePeriod=30
Jul 02 21:36:27.820620 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.820414    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="agent" containerID="cri-o://ef4d8b081cbfabd567d2c4d8b30b22c7dfd68e8a48642c942e9133b5582c93ed" gracePeriod=30
Jul 02 21:36:27.821260 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.820420    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kube-rbac-proxy" containerID="cri-o://421ee2ed43358d7ba1568daf7c34b920ad40f23df33564f3ba6d55e3dbd18780" gracePeriod=30
Jul 02 21:36:27.822250 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.822229    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"]
Jul 02 21:36:27.868700 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.868665    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:36:27.872962 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.872938    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"]
Jul 02 21:36:27.873202 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.873164    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52" podUID="ceb934c5-0868-411c-bb37-3b3f3f2bf665" containerName="kserve-container" containerID="cri-o://c3d483ef0f9b8f7ad0442d072319fc4b08bf71e204160a790457b324bfcfc917" gracePeriod=30
Jul 02 21:36:27.873264 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.873227    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52" podUID="ceb934c5-0868-411c-bb37-3b3f3f2bf665" containerName="kube-rbac-proxy" containerID="cri-o://72f1b7dbe99c9e06f6e15fda439af48d626cbee16518a158801811c09cc972fd" gracePeriod=30
Jul 02 21:36:27.873699 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.873677    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a693b634-4747-43d7-8648-ff8d2843ad6d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb\" (UID: \"a693b634-4747-43d7-8648-ff8d2843ad6d\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:36:27.873771 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.873720    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a693b634-4747-43d7-8648-ff8d2843ad6d-proxy-tls\") pod \"isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb\" (UID: \"a693b634-4747-43d7-8648-ff8d2843ad6d\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:36:27.873820 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.873800    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v744\" (UniqueName: \"kubernetes.io/projected/a693b634-4747-43d7-8648-ff8d2843ad6d-kube-api-access-8v744\") pod \"isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb\" (UID: \"a693b634-4747-43d7-8648-ff8d2843ad6d\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:36:27.873957 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.873933    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a693b634-4747-43d7-8648-ff8d2843ad6d-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb\" (UID: \"a693b634-4747-43d7-8648-ff8d2843ad6d\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:36:27.874020 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.873981    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a693b634-4747-43d7-8648-ff8d2843ad6d-cabundle-cert\") pod \"isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb\" (UID: \"a693b634-4747-43d7-8648-ff8d2843ad6d\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:36:27.928546 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.928524    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:36:27.974624 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.974597    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a693b634-4747-43d7-8648-ff8d2843ad6d-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb\" (UID: \"a693b634-4747-43d7-8648-ff8d2843ad6d\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:36:27.974744 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.974631    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a693b634-4747-43d7-8648-ff8d2843ad6d-cabundle-cert\") pod \"isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb\" (UID: \"a693b634-4747-43d7-8648-ff8d2843ad6d\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:36:27.974816 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.974756    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a693b634-4747-43d7-8648-ff8d2843ad6d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb\" (UID: \"a693b634-4747-43d7-8648-ff8d2843ad6d\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:36:27.974816 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.974794    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a693b634-4747-43d7-8648-ff8d2843ad6d-proxy-tls\") pod \"isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb\" (UID: \"a693b634-4747-43d7-8648-ff8d2843ad6d\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:36:27.974925 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.974830    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v744\" (UniqueName: \"kubernetes.io/projected/a693b634-4747-43d7-8648-ff8d2843ad6d-kube-api-access-8v744\") pod \"isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb\" (UID: \"a693b634-4747-43d7-8648-ff8d2843ad6d\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:36:27.975000 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:36:27.974927    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-serving-cert: secret "isvc-sklearn-scale-raw-e3b87-predictor-serving-cert" not found
Jul 02 21:36:27.975000 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:36:27.974990    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a693b634-4747-43d7-8648-ff8d2843ad6d-proxy-tls podName:a693b634-4747-43d7-8648-ff8d2843ad6d nodeName:}" failed. No retries permitted until 2026-07-02 21:36:28.474970914 +0000 UTC m=+1060.101472782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a693b634-4747-43d7-8648-ff8d2843ad6d-proxy-tls") pod "isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" (UID: "a693b634-4747-43d7-8648-ff8d2843ad6d") : secret "isvc-sklearn-scale-raw-e3b87-predictor-serving-cert" not found
Jul 02 21:36:27.975127 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.975028    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a693b634-4747-43d7-8648-ff8d2843ad6d-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb\" (UID: \"a693b634-4747-43d7-8648-ff8d2843ad6d\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:36:27.975325 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.975309    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a693b634-4747-43d7-8648-ff8d2843ad6d-cabundle-cert\") pod \"isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb\" (UID: \"a693b634-4747-43d7-8648-ff8d2843ad6d\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:36:27.975436 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.975416    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a693b634-4747-43d7-8648-ff8d2843ad6d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb\" (UID: \"a693b634-4747-43d7-8648-ff8d2843ad6d\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:36:27.983561 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:27.983537    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v744\" (UniqueName: \"kubernetes.io/projected/a693b634-4747-43d7-8648-ff8d2843ad6d-kube-api-access-8v744\") pod \"isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb\" (UID: \"a693b634-4747-43d7-8648-ff8d2843ad6d\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:36:28.114417 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.114388    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"
Jul 02 21:36:28.117398 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.117372    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:36:28.177164 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.177130    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ceb934c5-0868-411c-bb37-3b3f3f2bf665-kube-rbac-proxy-sar-config\") pod \"ceb934c5-0868-411c-bb37-3b3f3f2bf665\" (UID: \"ceb934c5-0868-411c-bb37-3b3f3f2bf665\") "
Jul 02 21:36:28.177324 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.177231    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceb934c5-0868-411c-bb37-3b3f3f2bf665-proxy-tls\") pod \"ceb934c5-0868-411c-bb37-3b3f3f2bf665\" (UID: \"ceb934c5-0868-411c-bb37-3b3f3f2bf665\") "
Jul 02 21:36:28.177324 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.177262    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbsnb\" (UniqueName: \"kubernetes.io/projected/ceb934c5-0868-411c-bb37-3b3f3f2bf665-kube-api-access-mbsnb\") pod \"ceb934c5-0868-411c-bb37-3b3f3f2bf665\" (UID: \"ceb934c5-0868-411c-bb37-3b3f3f2bf665\") "
Jul 02 21:36:28.177465 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.177444    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceb934c5-0868-411c-bb37-3b3f3f2bf665-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ceb934c5-0868-411c-bb37-3b3f3f2bf665" (UID: "ceb934c5-0868-411c-bb37-3b3f3f2bf665"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:36:28.177543 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.177530    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ceb934c5-0868-411c-bb37-3b3f3f2bf665-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:36:28.179103 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.179078    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb934c5-0868-411c-bb37-3b3f3f2bf665-kube-api-access-mbsnb" (OuterVolumeSpecName: "kube-api-access-mbsnb") pod "ceb934c5-0868-411c-bb37-3b3f3f2bf665" (UID: "ceb934c5-0868-411c-bb37-3b3f3f2bf665"). InnerVolumeSpecName "kube-api-access-mbsnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:36:28.179103 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.179085    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceb934c5-0868-411c-bb37-3b3f3f2bf665-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ceb934c5-0868-411c-bb37-3b3f3f2bf665" (UID: "ceb934c5-0868-411c-bb37-3b3f3f2bf665"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:36:28.278100 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.278074    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceb934c5-0868-411c-bb37-3b3f3f2bf665-proxy-tls\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:36:28.278100 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.278096    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbsnb\" (UniqueName: \"kubernetes.io/projected/ceb934c5-0868-411c-bb37-3b3f3f2bf665-kube-api-access-mbsnb\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:36:28.403365 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.403332    2575 generic.go:356] "Generic (PLEG): container finished" podID="ceb934c5-0868-411c-bb37-3b3f3f2bf665" containerID="72f1b7dbe99c9e06f6e15fda439af48d626cbee16518a158801811c09cc972fd" exitCode=2
Jul 02 21:36:28.403365 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.403363    2575 generic.go:356] "Generic (PLEG): container finished" podID="ceb934c5-0868-411c-bb37-3b3f3f2bf665" containerID="c3d483ef0f9b8f7ad0442d072319fc4b08bf71e204160a790457b324bfcfc917" exitCode=2
Jul 02 21:36:28.403598 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.403385    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"
Jul 02 21:36:28.403598 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.403423    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52" event={"ID":"ceb934c5-0868-411c-bb37-3b3f3f2bf665","Type":"ContainerDied","Data":"72f1b7dbe99c9e06f6e15fda439af48d626cbee16518a158801811c09cc972fd"}
Jul 02 21:36:28.403598 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.403490    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52" event={"ID":"ceb934c5-0868-411c-bb37-3b3f3f2bf665","Type":"ContainerDied","Data":"c3d483ef0f9b8f7ad0442d072319fc4b08bf71e204160a790457b324bfcfc917"}
Jul 02 21:36:28.403598 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.403507    2575 scope.go:117] "RemoveContainer" containerID="72f1b7dbe99c9e06f6e15fda439af48d626cbee16518a158801811c09cc972fd"
Jul 02 21:36:28.403598 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.403521    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52" event={"ID":"ceb934c5-0868-411c-bb37-3b3f3f2bf665","Type":"ContainerDied","Data":"1cc2a0c249c7c8f8101214de1e2f8b5ea8b0abeced64f77e10ddfa0627378e1c"}
Jul 02 21:36:28.406163 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.406143    2575 generic.go:356] "Generic (PLEG): container finished" podID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerID="421ee2ed43358d7ba1568daf7c34b920ad40f23df33564f3ba6d55e3dbd18780" exitCode=2
Jul 02 21:36:28.406293 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.406215    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" event={"ID":"3aadcb6a-96b7-456e-8eb7-2b710f299456","Type":"ContainerDied","Data":"421ee2ed43358d7ba1568daf7c34b920ad40f23df33564f3ba6d55e3dbd18780"}
Jul 02 21:36:28.412209 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.412187    2575 scope.go:117] "RemoveContainer" containerID="c3d483ef0f9b8f7ad0442d072319fc4b08bf71e204160a790457b324bfcfc917"
Jul 02 21:36:28.419377 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.419357    2575 scope.go:117] "RemoveContainer" containerID="72f1b7dbe99c9e06f6e15fda439af48d626cbee16518a158801811c09cc972fd"
Jul 02 21:36:28.419629 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:36:28.419612    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f1b7dbe99c9e06f6e15fda439af48d626cbee16518a158801811c09cc972fd\": container with ID starting with 72f1b7dbe99c9e06f6e15fda439af48d626cbee16518a158801811c09cc972fd not found: ID does not exist" containerID="72f1b7dbe99c9e06f6e15fda439af48d626cbee16518a158801811c09cc972fd"
Jul 02 21:36:28.419683 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.419636    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f1b7dbe99c9e06f6e15fda439af48d626cbee16518a158801811c09cc972fd"} err="failed to get container status \"72f1b7dbe99c9e06f6e15fda439af48d626cbee16518a158801811c09cc972fd\": rpc error: code = NotFound desc = could not find container \"72f1b7dbe99c9e06f6e15fda439af48d626cbee16518a158801811c09cc972fd\": container with ID starting with 72f1b7dbe99c9e06f6e15fda439af48d626cbee16518a158801811c09cc972fd not found: ID does not exist"
Jul 02 21:36:28.419683 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.419652    2575 scope.go:117] "RemoveContainer" containerID="c3d483ef0f9b8f7ad0442d072319fc4b08bf71e204160a790457b324bfcfc917"
Jul 02 21:36:28.419885 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:36:28.419869    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3d483ef0f9b8f7ad0442d072319fc4b08bf71e204160a790457b324bfcfc917\": container with ID starting with c3d483ef0f9b8f7ad0442d072319fc4b08bf71e204160a790457b324bfcfc917 not found: ID does not exist" containerID="c3d483ef0f9b8f7ad0442d072319fc4b08bf71e204160a790457b324bfcfc917"
Jul 02 21:36:28.419934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.419894    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d483ef0f9b8f7ad0442d072319fc4b08bf71e204160a790457b324bfcfc917"} err="failed to get container status \"c3d483ef0f9b8f7ad0442d072319fc4b08bf71e204160a790457b324bfcfc917\": rpc error: code = NotFound desc = could not find container \"c3d483ef0f9b8f7ad0442d072319fc4b08bf71e204160a790457b324bfcfc917\": container with ID starting with c3d483ef0f9b8f7ad0442d072319fc4b08bf71e204160a790457b324bfcfc917 not found: ID does not exist"
Jul 02 21:36:28.419934 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.419909    2575 scope.go:117] "RemoveContainer" containerID="72f1b7dbe99c9e06f6e15fda439af48d626cbee16518a158801811c09cc972fd"
Jul 02 21:36:28.420143 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.420122    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f1b7dbe99c9e06f6e15fda439af48d626cbee16518a158801811c09cc972fd"} err="failed to get container status \"72f1b7dbe99c9e06f6e15fda439af48d626cbee16518a158801811c09cc972fd\": rpc error: code = NotFound desc = could not find container \"72f1b7dbe99c9e06f6e15fda439af48d626cbee16518a158801811c09cc972fd\": container with ID starting with 72f1b7dbe99c9e06f6e15fda439af48d626cbee16518a158801811c09cc972fd not found: ID does not exist"
Jul 02 21:36:28.420225 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.420145    2575 scope.go:117] "RemoveContainer" containerID="c3d483ef0f9b8f7ad0442d072319fc4b08bf71e204160a790457b324bfcfc917"
Jul 02 21:36:28.420525 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.420507    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d483ef0f9b8f7ad0442d072319fc4b08bf71e204160a790457b324bfcfc917"} err="failed to get container status \"c3d483ef0f9b8f7ad0442d072319fc4b08bf71e204160a790457b324bfcfc917\": rpc error: code = NotFound desc = could not find container \"c3d483ef0f9b8f7ad0442d072319fc4b08bf71e204160a790457b324bfcfc917\": container with ID starting with c3d483ef0f9b8f7ad0442d072319fc4b08bf71e204160a790457b324bfcfc917 not found: ID does not exist"
Jul 02 21:36:28.424802 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.424783    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"]
Jul 02 21:36:28.428304 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.428284    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1f359-predictor-864d8c9886-4cb52"]
Jul 02 21:36:28.479753 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.479706    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a693b634-4747-43d7-8648-ff8d2843ad6d-proxy-tls\") pod \"isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb\" (UID: \"a693b634-4747-43d7-8648-ff8d2843ad6d\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:36:28.482108 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.482082    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a693b634-4747-43d7-8648-ff8d2843ad6d-proxy-tls\") pod \"isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb\" (UID: \"a693b634-4747-43d7-8648-ff8d2843ad6d\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:36:28.723239 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.723128    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:36:28.852602 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:28.852578    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"]
Jul 02 21:36:28.854742 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:36:28.854716    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda693b634_4747_43d7_8648_ff8d2843ad6d.slice/crio-a187e1f4bbce035c6c7d8107b13bd1d521381d05c35a6079cba041c598b30ed3 WatchSource:0}: Error finding container a187e1f4bbce035c6c7d8107b13bd1d521381d05c35a6079cba041c598b30ed3: Status 404 returned error can't find the container with id a187e1f4bbce035c6c7d8107b13bd1d521381d05c35a6079cba041c598b30ed3
Jul 02 21:36:29.014867 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:29.014792    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceb934c5-0868-411c-bb37-3b3f3f2bf665" path="/var/lib/kubelet/pods/ceb934c5-0868-411c-bb37-3b3f3f2bf665/volumes"
Jul 02 21:36:29.411355 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:29.411318    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" event={"ID":"a693b634-4747-43d7-8648-ff8d2843ad6d","Type":"ContainerStarted","Data":"fb118565c515ea74af80b96aa0ee49207b59e987eaafdb619ed786769fb5b989"}
Jul 02 21:36:29.411545 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:29.411389    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" event={"ID":"a693b634-4747-43d7-8648-ff8d2843ad6d","Type":"ContainerStarted","Data":"a187e1f4bbce035c6c7d8107b13bd1d521381d05c35a6079cba041c598b30ed3"}
Jul 02 21:36:30.048983 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:30.048941    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 02 21:36:32.426164 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:32.426134    2575 generic.go:356] "Generic (PLEG): container finished" podID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerID="a2049dfe85af2959d25aaa96f7b82ff9e6f7078a19da9e851eb1887542a443d9" exitCode=0
Jul 02 21:36:32.426574 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:32.426204    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" event={"ID":"3aadcb6a-96b7-456e-8eb7-2b710f299456","Type":"ContainerDied","Data":"a2049dfe85af2959d25aaa96f7b82ff9e6f7078a19da9e851eb1887542a443d9"}
Jul 02 21:36:33.009688 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:33.009641    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 21:36:33.009996 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:33.009971    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:36:33.431871 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:33.431841    2575 generic.go:356] "Generic (PLEG): container finished" podID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerID="fb118565c515ea74af80b96aa0ee49207b59e987eaafdb619ed786769fb5b989" exitCode=0
Jul 02 21:36:33.432254 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:33.431914    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" event={"ID":"a693b634-4747-43d7-8648-ff8d2843ad6d","Type":"ContainerDied","Data":"fb118565c515ea74af80b96aa0ee49207b59e987eaafdb619ed786769fb5b989"}
Jul 02 21:36:34.443590 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:34.443553    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" event={"ID":"a693b634-4747-43d7-8648-ff8d2843ad6d","Type":"ContainerStarted","Data":"283f5ee2ee703bf885b276471f26cb80e25738f61315c7fc9ff35ae4804dc517"}
Jul 02 21:36:34.443969 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:34.443635    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" event={"ID":"a693b634-4747-43d7-8648-ff8d2843ad6d","Type":"ContainerStarted","Data":"ea9e0c70d74930bc37e01d96826cdaa2833175133da468eb57f9bb80e942fd22"}
Jul 02 21:36:34.443969 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:34.443927    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:36:34.444101 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:34.444081    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:36:34.445334 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:34.445310    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 21:36:34.468052 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:34.468011    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podStartSLOduration=3.935115975 podStartE2EDuration="7.467999626s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.532883651s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:36:34.467619606 +0000 UTC m=+1066.094121503" watchObservedRunningTime="2026-07-02 21:36:34.467999626 +0000 UTC m=+1066.094501502"
Jul 02 21:36:35.049511 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:35.049470    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 02 21:36:35.447585 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:35.447553    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 21:36:36.451253 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:36.451211    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 21:36:40.049574 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:40.049534    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 02 21:36:40.050080 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:40.049652    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:36:41.455448 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:41.455376    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:36:41.456014 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:41.455986    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 21:36:43.009785 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:43.009731    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 21:36:43.010267 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:43.010041    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:36:45.049094 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:45.049054    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 02 21:36:50.048940 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:50.048894    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 02 21:36:51.455826 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:51.455788    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 21:36:53.009713 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:53.009663    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 21:36:53.010090 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:53.009920    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 21:36:53.015164 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:53.015145    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:36:53.015276 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:53.015266    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:36:55.049028 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:55.048987    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 02 21:36:57.975766 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:57.975740    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:36:58.005831 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.005799    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3aadcb6a-96b7-456e-8eb7-2b710f299456-proxy-tls\") pod \"3aadcb6a-96b7-456e-8eb7-2b710f299456\" (UID: \"3aadcb6a-96b7-456e-8eb7-2b710f299456\") "
Jul 02 21:36:58.005992 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.005903    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3aadcb6a-96b7-456e-8eb7-2b710f299456-kserve-provision-location\") pod \"3aadcb6a-96b7-456e-8eb7-2b710f299456\" (UID: \"3aadcb6a-96b7-456e-8eb7-2b710f299456\") "
Jul 02 21:36:58.005992 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.005955    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3aadcb6a-96b7-456e-8eb7-2b710f299456-cabundle-cert\") pod \"3aadcb6a-96b7-456e-8eb7-2b710f299456\" (UID: \"3aadcb6a-96b7-456e-8eb7-2b710f299456\") "
Jul 02 21:36:58.005992 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.005984    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3aadcb6a-96b7-456e-8eb7-2b710f299456-kube-rbac-proxy-sar-config\") pod \"3aadcb6a-96b7-456e-8eb7-2b710f299456\" (UID: \"3aadcb6a-96b7-456e-8eb7-2b710f299456\") "
Jul 02 21:36:58.006124 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.006017    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5thtp\" (UniqueName: \"kubernetes.io/projected/3aadcb6a-96b7-456e-8eb7-2b710f299456-kube-api-access-5thtp\") pod \"3aadcb6a-96b7-456e-8eb7-2b710f299456\" (UID: \"3aadcb6a-96b7-456e-8eb7-2b710f299456\") "
Jul 02 21:36:58.006317 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.006289    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aadcb6a-96b7-456e-8eb7-2b710f299456-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3aadcb6a-96b7-456e-8eb7-2b710f299456" (UID: "3aadcb6a-96b7-456e-8eb7-2b710f299456"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:36:58.006408 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.006386    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aadcb6a-96b7-456e-8eb7-2b710f299456-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3aadcb6a-96b7-456e-8eb7-2b710f299456" (UID: "3aadcb6a-96b7-456e-8eb7-2b710f299456"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:36:58.006462 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.006394    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aadcb6a-96b7-456e-8eb7-2b710f299456-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3aadcb6a-96b7-456e-8eb7-2b710f299456" (UID: "3aadcb6a-96b7-456e-8eb7-2b710f299456"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:36:58.007997 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.007972    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aadcb6a-96b7-456e-8eb7-2b710f299456-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3aadcb6a-96b7-456e-8eb7-2b710f299456" (UID: "3aadcb6a-96b7-456e-8eb7-2b710f299456"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:36:58.008049 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.008017    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aadcb6a-96b7-456e-8eb7-2b710f299456-kube-api-access-5thtp" (OuterVolumeSpecName: "kube-api-access-5thtp") pod "3aadcb6a-96b7-456e-8eb7-2b710f299456" (UID: "3aadcb6a-96b7-456e-8eb7-2b710f299456"). InnerVolumeSpecName "kube-api-access-5thtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:36:58.107458 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.107371    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3aadcb6a-96b7-456e-8eb7-2b710f299456-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:36:58.107458 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.107408    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5thtp\" (UniqueName: \"kubernetes.io/projected/3aadcb6a-96b7-456e-8eb7-2b710f299456-kube-api-access-5thtp\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:36:58.107458 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.107419    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3aadcb6a-96b7-456e-8eb7-2b710f299456-proxy-tls\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:36:58.107458 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.107428    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3aadcb6a-96b7-456e-8eb7-2b710f299456-kserve-provision-location\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:36:58.107458 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.107440    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3aadcb6a-96b7-456e-8eb7-2b710f299456-cabundle-cert\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:36:58.539355 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.539316    2575 generic.go:356] "Generic (PLEG): container finished" podID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerID="ef4d8b081cbfabd567d2c4d8b30b22c7dfd68e8a48642c942e9133b5582c93ed" exitCode=0
Jul 02 21:36:58.539593 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.539386    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" event={"ID":"3aadcb6a-96b7-456e-8eb7-2b710f299456","Type":"ContainerDied","Data":"ef4d8b081cbfabd567d2c4d8b30b22c7dfd68e8a48642c942e9133b5582c93ed"}
Jul 02 21:36:58.539593 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.539448    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6" event={"ID":"3aadcb6a-96b7-456e-8eb7-2b710f299456","Type":"ContainerDied","Data":"cf5a60137314c14378d74151210b4134980869a2b118b4b9e67aeca5b84c39e1"}
Jul 02 21:36:58.539593 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.539392    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"
Jul 02 21:36:58.539593 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.539486    2575 scope.go:117] "RemoveContainer" containerID="ef4d8b081cbfabd567d2c4d8b30b22c7dfd68e8a48642c942e9133b5582c93ed"
Jul 02 21:36:58.547986 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.547964    2575 scope.go:117] "RemoveContainer" containerID="421ee2ed43358d7ba1568daf7c34b920ad40f23df33564f3ba6d55e3dbd18780"
Jul 02 21:36:58.555581 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.555560    2575 scope.go:117] "RemoveContainer" containerID="a2049dfe85af2959d25aaa96f7b82ff9e6f7078a19da9e851eb1887542a443d9"
Jul 02 21:36:58.562753 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.562731    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"]
Jul 02 21:36:58.563269 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.563251    2575 scope.go:117] "RemoveContainer" containerID="d3d9dfc42d32cc49db285a11fa274c803418a007cfc7a7a9e86aede683a26fac"
Jul 02 21:36:58.565872 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.565847    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1f359-predictor-5778d849d6-2mwz6"]
Jul 02 21:36:58.570727 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.570710    2575 scope.go:117] "RemoveContainer" containerID="ef4d8b081cbfabd567d2c4d8b30b22c7dfd68e8a48642c942e9133b5582c93ed"
Jul 02 21:36:58.571001 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:36:58.570979    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef4d8b081cbfabd567d2c4d8b30b22c7dfd68e8a48642c942e9133b5582c93ed\": container with ID starting with ef4d8b081cbfabd567d2c4d8b30b22c7dfd68e8a48642c942e9133b5582c93ed not found: ID does not exist" containerID="ef4d8b081cbfabd567d2c4d8b30b22c7dfd68e8a48642c942e9133b5582c93ed"
Jul 02 21:36:58.571105 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.571006    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef4d8b081cbfabd567d2c4d8b30b22c7dfd68e8a48642c942e9133b5582c93ed"} err="failed to get container status \"ef4d8b081cbfabd567d2c4d8b30b22c7dfd68e8a48642c942e9133b5582c93ed\": rpc error: code = NotFound desc = could not find container \"ef4d8b081cbfabd567d2c4d8b30b22c7dfd68e8a48642c942e9133b5582c93ed\": container with ID starting with ef4d8b081cbfabd567d2c4d8b30b22c7dfd68e8a48642c942e9133b5582c93ed not found: ID does not exist"
Jul 02 21:36:58.571105 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.571024    2575 scope.go:117] "RemoveContainer" containerID="421ee2ed43358d7ba1568daf7c34b920ad40f23df33564f3ba6d55e3dbd18780"
Jul 02 21:36:58.571268 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:36:58.571251    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"421ee2ed43358d7ba1568daf7c34b920ad40f23df33564f3ba6d55e3dbd18780\": container with ID starting with 421ee2ed43358d7ba1568daf7c34b920ad40f23df33564f3ba6d55e3dbd18780 not found: ID does not exist" containerID="421ee2ed43358d7ba1568daf7c34b920ad40f23df33564f3ba6d55e3dbd18780"
Jul 02 21:36:58.571311 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.571274    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"421ee2ed43358d7ba1568daf7c34b920ad40f23df33564f3ba6d55e3dbd18780"} err="failed to get container status \"421ee2ed43358d7ba1568daf7c34b920ad40f23df33564f3ba6d55e3dbd18780\": rpc error: code = NotFound desc = could not find container \"421ee2ed43358d7ba1568daf7c34b920ad40f23df33564f3ba6d55e3dbd18780\": container with ID starting with 421ee2ed43358d7ba1568daf7c34b920ad40f23df33564f3ba6d55e3dbd18780 not found: ID does not exist"
Jul 02 21:36:58.571311 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.571290    2575 scope.go:117] "RemoveContainer" containerID="a2049dfe85af2959d25aaa96f7b82ff9e6f7078a19da9e851eb1887542a443d9"
Jul 02 21:36:58.571521 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:36:58.571503    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2049dfe85af2959d25aaa96f7b82ff9e6f7078a19da9e851eb1887542a443d9\": container with ID starting with a2049dfe85af2959d25aaa96f7b82ff9e6f7078a19da9e851eb1887542a443d9 not found: ID does not exist" containerID="a2049dfe85af2959d25aaa96f7b82ff9e6f7078a19da9e851eb1887542a443d9"
Jul 02 21:36:58.571585 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.571529    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2049dfe85af2959d25aaa96f7b82ff9e6f7078a19da9e851eb1887542a443d9"} err="failed to get container status \"a2049dfe85af2959d25aaa96f7b82ff9e6f7078a19da9e851eb1887542a443d9\": rpc error: code = NotFound desc = could not find container \"a2049dfe85af2959d25aaa96f7b82ff9e6f7078a19da9e851eb1887542a443d9\": container with ID starting with a2049dfe85af2959d25aaa96f7b82ff9e6f7078a19da9e851eb1887542a443d9 not found: ID does not exist"
Jul 02 21:36:58.571585 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.571551    2575 scope.go:117] "RemoveContainer" containerID="d3d9dfc42d32cc49db285a11fa274c803418a007cfc7a7a9e86aede683a26fac"
Jul 02 21:36:58.571818 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:36:58.571800    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3d9dfc42d32cc49db285a11fa274c803418a007cfc7a7a9e86aede683a26fac\": container with ID starting with d3d9dfc42d32cc49db285a11fa274c803418a007cfc7a7a9e86aede683a26fac not found: ID does not exist" containerID="d3d9dfc42d32cc49db285a11fa274c803418a007cfc7a7a9e86aede683a26fac"
Jul 02 21:36:58.571859 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:58.571825    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3d9dfc42d32cc49db285a11fa274c803418a007cfc7a7a9e86aede683a26fac"} err="failed to get container status \"d3d9dfc42d32cc49db285a11fa274c803418a007cfc7a7a9e86aede683a26fac\": rpc error: code = NotFound desc = could not find container \"d3d9dfc42d32cc49db285a11fa274c803418a007cfc7a7a9e86aede683a26fac\": container with ID starting with d3d9dfc42d32cc49db285a11fa274c803418a007cfc7a7a9e86aede683a26fac not found: ID does not exist"
Jul 02 21:36:59.014645 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:36:59.014615    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" path="/var/lib/kubelet/pods/3aadcb6a-96b7-456e-8eb7-2b710f299456/volumes"
Jul 02 21:37:01.456044 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:37:01.456009    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 21:37:07.769097 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:37:07.769055    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:37:11.456152 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:37:11.456113    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 21:37:21.455771 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:37:21.455732    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 21:37:31.455804 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:37:31.455761    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 21:37:41.456351 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:37:41.456311    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 21:37:51.456347 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:37:51.456310    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 21:37:52.009649 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:37:52.009612    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 21:38:02.009708 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:38:02.009663    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 21:38:12.010027 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:38:12.009939    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 21:38:22.010017 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:38:22.009975    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 21:38:32.010382 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:38:32.010337    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 21:38:42.009810 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:38:42.009738    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 21:38:52.010082 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:38:52.010045    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 21:38:57.070060 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:38:57.070028    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:39:02.010222 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:02.010154    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:39:08.013838 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:08.013808    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"]
Jul 02 21:39:08.014228 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:08.014051    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container" containerID="cri-o://ea9e0c70d74930bc37e01d96826cdaa2833175133da468eb57f9bb80e942fd22" gracePeriod=30
Jul 02 21:39:08.014228 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:08.014099    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kube-rbac-proxy" containerID="cri-o://283f5ee2ee703bf885b276471f26cb80e25738f61315c7fc9ff35ae4804dc517" gracePeriod=30
Jul 02 21:39:08.023218 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:08.023189    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:39:09.054025 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:09.053995    2575 generic.go:356] "Generic (PLEG): container finished" podID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerID="283f5ee2ee703bf885b276471f26cb80e25738f61315c7fc9ff35ae4804dc517" exitCode=2
Jul 02 21:39:09.054393 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:09.054052    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" event={"ID":"a693b634-4747-43d7-8648-ff8d2843ad6d","Type":"ContainerDied","Data":"283f5ee2ee703bf885b276471f26cb80e25738f61315c7fc9ff35ae4804dc517"}
Jul 02 21:39:11.451881 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:11.451838    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 02 21:39:12.010324 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:12.010279    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 21:39:13.094038 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.094002    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:39:13.151159 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.151123    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"]
Jul 02 21:39:13.151610 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.151595    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kube-rbac-proxy"
Jul 02 21:39:13.151661 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.151613    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kube-rbac-proxy"
Jul 02 21:39:13.151661 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.151646    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="storage-initializer"
Jul 02 21:39:13.151661 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.151654    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="storage-initializer"
Jul 02 21:39:13.151760 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.151668    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kserve-container"
Jul 02 21:39:13.151760 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.151676    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kserve-container"
Jul 02 21:39:13.151760 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.151685    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb934c5-0868-411c-bb37-3b3f3f2bf665" containerName="kube-rbac-proxy"
Jul 02 21:39:13.151760 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.151693    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb934c5-0868-411c-bb37-3b3f3f2bf665" containerName="kube-rbac-proxy"
Jul 02 21:39:13.151760 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.151704    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb934c5-0868-411c-bb37-3b3f3f2bf665" containerName="kserve-container"
Jul 02 21:39:13.151760 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.151713    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb934c5-0868-411c-bb37-3b3f3f2bf665" containerName="kserve-container"
Jul 02 21:39:13.151760 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.151729    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="agent"
Jul 02 21:39:13.151760 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.151736    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="agent"
Jul 02 21:39:13.151997 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.151797    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kserve-container"
Jul 02 21:39:13.151997 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.151808    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="kube-rbac-proxy"
Jul 02 21:39:13.151997 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.151819    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aadcb6a-96b7-456e-8eb7-2b710f299456" containerName="agent"
Jul 02 21:39:13.151997 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.151830    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb934c5-0868-411c-bb37-3b3f3f2bf665" containerName="kserve-container"
Jul 02 21:39:13.151997 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.151839    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb934c5-0868-411c-bb37-3b3f3f2bf665" containerName="kube-rbac-proxy"
Jul 02 21:39:13.159527 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.159489    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:39:13.162819 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.162787    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-0bbdb6-kube-rbac-proxy-sar-config\""
Jul 02 21:39:13.163037 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.163018    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-0bbdb6-predictor-serving-cert\""
Jul 02 21:39:13.164771 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.164748    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"]
Jul 02 21:39:13.197291 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.197261    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rr46\" (UniqueName: \"kubernetes.io/projected/6394bdea-c72b-4f71-878f-10c98235628a-kube-api-access-7rr46\") pod \"isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9\" (UID: \"6394bdea-c72b-4f71-878f-10c98235628a\") " pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:39:13.197407 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.197301    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6394bdea-c72b-4f71-878f-10c98235628a-proxy-tls\") pod \"isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9\" (UID: \"6394bdea-c72b-4f71-878f-10c98235628a\") " pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:39:13.197407 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.197387    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6394bdea-c72b-4f71-878f-10c98235628a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9\" (UID: \"6394bdea-c72b-4f71-878f-10c98235628a\") " pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:39:13.197498 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.197416    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6394bdea-c72b-4f71-878f-10c98235628a-kserve-provision-location\") pod \"isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9\" (UID: \"6394bdea-c72b-4f71-878f-10c98235628a\") " pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:39:13.197498 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.197433    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6394bdea-c72b-4f71-878f-10c98235628a-cabundle-cert\") pod \"isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9\" (UID: \"6394bdea-c72b-4f71-878f-10c98235628a\") " pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:39:13.269448 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.269400    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:39:13.298695 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.298665    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6394bdea-c72b-4f71-878f-10c98235628a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9\" (UID: \"6394bdea-c72b-4f71-878f-10c98235628a\") " pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:39:13.298820 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.298706    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6394bdea-c72b-4f71-878f-10c98235628a-kserve-provision-location\") pod \"isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9\" (UID: \"6394bdea-c72b-4f71-878f-10c98235628a\") " pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:39:13.298892 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.298829    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6394bdea-c72b-4f71-878f-10c98235628a-cabundle-cert\") pod \"isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9\" (UID: \"6394bdea-c72b-4f71-878f-10c98235628a\") " pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:39:13.298949 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.298895    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rr46\" (UniqueName: \"kubernetes.io/projected/6394bdea-c72b-4f71-878f-10c98235628a-kube-api-access-7rr46\") pod \"isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9\" (UID: \"6394bdea-c72b-4f71-878f-10c98235628a\") " pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:39:13.299005 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.298958    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6394bdea-c72b-4f71-878f-10c98235628a-proxy-tls\") pod \"isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9\" (UID: \"6394bdea-c72b-4f71-878f-10c98235628a\") " pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:39:13.299128 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.299109    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6394bdea-c72b-4f71-878f-10c98235628a-kserve-provision-location\") pod \"isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9\" (UID: \"6394bdea-c72b-4f71-878f-10c98235628a\") " pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:39:13.299463 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.299445    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6394bdea-c72b-4f71-878f-10c98235628a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9\" (UID: \"6394bdea-c72b-4f71-878f-10c98235628a\") " pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:39:13.299525 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.299452    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6394bdea-c72b-4f71-878f-10c98235628a-cabundle-cert\") pod \"isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9\" (UID: \"6394bdea-c72b-4f71-878f-10c98235628a\") " pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:39:13.301456 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.301436    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6394bdea-c72b-4f71-878f-10c98235628a-proxy-tls\") pod \"isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9\" (UID: \"6394bdea-c72b-4f71-878f-10c98235628a\") " pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:39:13.306560 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.306540    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rr46\" (UniqueName: \"kubernetes.io/projected/6394bdea-c72b-4f71-878f-10c98235628a-kube-api-access-7rr46\") pod \"isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9\" (UID: \"6394bdea-c72b-4f71-878f-10c98235628a\") " pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:39:13.472667 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.472634    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:39:13.604087 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:13.604061    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"]
Jul 02 21:39:13.606131 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:39:13.606101    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6394bdea_c72b_4f71_878f_10c98235628a.slice/crio-20c9645279014e42799c56c78a83a6bac09f405f148320d18028bb5c69549f21 WatchSource:0}: Error finding container 20c9645279014e42799c56c78a83a6bac09f405f148320d18028bb5c69549f21: Status 404 returned error can't find the container with id 20c9645279014e42799c56c78a83a6bac09f405f148320d18028bb5c69549f21
Jul 02 21:39:14.076583 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:14.076543    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9" event={"ID":"6394bdea-c72b-4f71-878f-10c98235628a","Type":"ContainerStarted","Data":"edc27d0a6d1e3a061cf0b14971616dc52ceeb438e0889b8dde0e18e4fab3d524"}
Jul 02 21:39:14.076752 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:14.076604    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9" event={"ID":"6394bdea-c72b-4f71-878f-10c98235628a","Type":"ContainerStarted","Data":"20c9645279014e42799c56c78a83a6bac09f405f148320d18028bb5c69549f21"}
Jul 02 21:39:16.452498 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:16.452452    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 02 21:39:17.551761 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:17.551738    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:39:17.636846 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:17.636815    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8v744\" (UniqueName: \"kubernetes.io/projected/a693b634-4747-43d7-8648-ff8d2843ad6d-kube-api-access-8v744\") pod \"a693b634-4747-43d7-8648-ff8d2843ad6d\" (UID: \"a693b634-4747-43d7-8648-ff8d2843ad6d\") "
Jul 02 21:39:17.636986 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:17.636861    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a693b634-4747-43d7-8648-ff8d2843ad6d-kserve-provision-location\") pod \"a693b634-4747-43d7-8648-ff8d2843ad6d\" (UID: \"a693b634-4747-43d7-8648-ff8d2843ad6d\") "
Jul 02 21:39:17.636986 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:17.636881    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a693b634-4747-43d7-8648-ff8d2843ad6d-cabundle-cert\") pod \"a693b634-4747-43d7-8648-ff8d2843ad6d\" (UID: \"a693b634-4747-43d7-8648-ff8d2843ad6d\") "
Jul 02 21:39:17.637093 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:17.636996    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a693b634-4747-43d7-8648-ff8d2843ad6d-kube-rbac-proxy-sar-config\") pod \"a693b634-4747-43d7-8648-ff8d2843ad6d\" (UID: \"a693b634-4747-43d7-8648-ff8d2843ad6d\") "
Jul 02 21:39:17.637093 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:17.637055    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a693b634-4747-43d7-8648-ff8d2843ad6d-proxy-tls\") pod \"a693b634-4747-43d7-8648-ff8d2843ad6d\" (UID: \"a693b634-4747-43d7-8648-ff8d2843ad6d\") "
Jul 02 21:39:17.637278 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:17.637186    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a693b634-4747-43d7-8648-ff8d2843ad6d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a693b634-4747-43d7-8648-ff8d2843ad6d" (UID: "a693b634-4747-43d7-8648-ff8d2843ad6d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:39:17.637278 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:17.637228    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a693b634-4747-43d7-8648-ff8d2843ad6d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a693b634-4747-43d7-8648-ff8d2843ad6d" (UID: "a693b634-4747-43d7-8648-ff8d2843ad6d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:39:17.637438 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:17.637332    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a693b634-4747-43d7-8648-ff8d2843ad6d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a693b634-4747-43d7-8648-ff8d2843ad6d" (UID: "a693b634-4747-43d7-8648-ff8d2843ad6d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:39:17.638992 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:17.638967    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a693b634-4747-43d7-8648-ff8d2843ad6d-kube-api-access-8v744" (OuterVolumeSpecName: "kube-api-access-8v744") pod "a693b634-4747-43d7-8648-ff8d2843ad6d" (UID: "a693b634-4747-43d7-8648-ff8d2843ad6d"). InnerVolumeSpecName "kube-api-access-8v744". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:39:17.639071 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:17.639026    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a693b634-4747-43d7-8648-ff8d2843ad6d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a693b634-4747-43d7-8648-ff8d2843ad6d" (UID: "a693b634-4747-43d7-8648-ff8d2843ad6d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:39:17.737841 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:17.737815    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v744\" (UniqueName: \"kubernetes.io/projected/a693b634-4747-43d7-8648-ff8d2843ad6d-kube-api-access-8v744\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:39:17.737841 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:17.737839    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a693b634-4747-43d7-8648-ff8d2843ad6d-kserve-provision-location\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:39:17.737841 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:17.737849    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a693b634-4747-43d7-8648-ff8d2843ad6d-cabundle-cert\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:39:17.738048 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:17.737858    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a693b634-4747-43d7-8648-ff8d2843ad6d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:39:17.738048 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:17.737867    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a693b634-4747-43d7-8648-ff8d2843ad6d-proxy-tls\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:39:18.094522 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:18.094481    2575 generic.go:356] "Generic (PLEG): container finished" podID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerID="ea9e0c70d74930bc37e01d96826cdaa2833175133da468eb57f9bb80e942fd22" exitCode=0
Jul 02 21:39:18.094705 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:18.094553    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"
Jul 02 21:39:18.094705 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:18.094564    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" event={"ID":"a693b634-4747-43d7-8648-ff8d2843ad6d","Type":"ContainerDied","Data":"ea9e0c70d74930bc37e01d96826cdaa2833175133da468eb57f9bb80e942fd22"}
Jul 02 21:39:18.094705 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:18.094633    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb" event={"ID":"a693b634-4747-43d7-8648-ff8d2843ad6d","Type":"ContainerDied","Data":"a187e1f4bbce035c6c7d8107b13bd1d521381d05c35a6079cba041c598b30ed3"}
Jul 02 21:39:18.094705 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:18.094662    2575 scope.go:117] "RemoveContainer" containerID="283f5ee2ee703bf885b276471f26cb80e25738f61315c7fc9ff35ae4804dc517"
Jul 02 21:39:18.096238 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:18.096215    2575 generic.go:356] "Generic (PLEG): container finished" podID="6394bdea-c72b-4f71-878f-10c98235628a" containerID="edc27d0a6d1e3a061cf0b14971616dc52ceeb438e0889b8dde0e18e4fab3d524" exitCode=0
Jul 02 21:39:18.096349 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:18.096286    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9" event={"ID":"6394bdea-c72b-4f71-878f-10c98235628a","Type":"ContainerDied","Data":"edc27d0a6d1e3a061cf0b14971616dc52ceeb438e0889b8dde0e18e4fab3d524"}
Jul 02 21:39:18.103228 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:18.103209    2575 scope.go:117] "RemoveContainer" containerID="ea9e0c70d74930bc37e01d96826cdaa2833175133da468eb57f9bb80e942fd22"
Jul 02 21:39:18.111072 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:18.111049    2575 scope.go:117] "RemoveContainer" containerID="fb118565c515ea74af80b96aa0ee49207b59e987eaafdb619ed786769fb5b989"
Jul 02 21:39:18.120278 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:18.120261    2575 scope.go:117] "RemoveContainer" containerID="283f5ee2ee703bf885b276471f26cb80e25738f61315c7fc9ff35ae4804dc517"
Jul 02 21:39:18.120535 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:39:18.120514    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"283f5ee2ee703bf885b276471f26cb80e25738f61315c7fc9ff35ae4804dc517\": container with ID starting with 283f5ee2ee703bf885b276471f26cb80e25738f61315c7fc9ff35ae4804dc517 not found: ID does not exist" containerID="283f5ee2ee703bf885b276471f26cb80e25738f61315c7fc9ff35ae4804dc517"
Jul 02 21:39:18.120626 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:18.120540    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"283f5ee2ee703bf885b276471f26cb80e25738f61315c7fc9ff35ae4804dc517"} err="failed to get container status \"283f5ee2ee703bf885b276471f26cb80e25738f61315c7fc9ff35ae4804dc517\": rpc error: code = NotFound desc = could not find container \"283f5ee2ee703bf885b276471f26cb80e25738f61315c7fc9ff35ae4804dc517\": container with ID starting with 283f5ee2ee703bf885b276471f26cb80e25738f61315c7fc9ff35ae4804dc517 not found: ID does not exist"
Jul 02 21:39:18.120626 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:18.120557    2575 scope.go:117] "RemoveContainer" containerID="ea9e0c70d74930bc37e01d96826cdaa2833175133da468eb57f9bb80e942fd22"
Jul 02 21:39:18.120773 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:39:18.120759    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea9e0c70d74930bc37e01d96826cdaa2833175133da468eb57f9bb80e942fd22\": container with ID starting with ea9e0c70d74930bc37e01d96826cdaa2833175133da468eb57f9bb80e942fd22 not found: ID does not exist" containerID="ea9e0c70d74930bc37e01d96826cdaa2833175133da468eb57f9bb80e942fd22"
Jul 02 21:39:18.120817 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:18.120781    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9e0c70d74930bc37e01d96826cdaa2833175133da468eb57f9bb80e942fd22"} err="failed to get container status \"ea9e0c70d74930bc37e01d96826cdaa2833175133da468eb57f9bb80e942fd22\": rpc error: code = NotFound desc = could not find container \"ea9e0c70d74930bc37e01d96826cdaa2833175133da468eb57f9bb80e942fd22\": container with ID starting with ea9e0c70d74930bc37e01d96826cdaa2833175133da468eb57f9bb80e942fd22 not found: ID does not exist"
Jul 02 21:39:18.120817 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:18.120793    2575 scope.go:117] "RemoveContainer" containerID="fb118565c515ea74af80b96aa0ee49207b59e987eaafdb619ed786769fb5b989"
Jul 02 21:39:18.120998 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:39:18.120981    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb118565c515ea74af80b96aa0ee49207b59e987eaafdb619ed786769fb5b989\": container with ID starting with fb118565c515ea74af80b96aa0ee49207b59e987eaafdb619ed786769fb5b989 not found: ID does not exist" containerID="fb118565c515ea74af80b96aa0ee49207b59e987eaafdb619ed786769fb5b989"
Jul 02 21:39:18.121043 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:18.121004    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb118565c515ea74af80b96aa0ee49207b59e987eaafdb619ed786769fb5b989"} err="failed to get container status \"fb118565c515ea74af80b96aa0ee49207b59e987eaafdb619ed786769fb5b989\": rpc error: code = NotFound desc = could not find container \"fb118565c515ea74af80b96aa0ee49207b59e987eaafdb619ed786769fb5b989\": container with ID starting with fb118565c515ea74af80b96aa0ee49207b59e987eaafdb619ed786769fb5b989 not found: ID does not exist"
Jul 02 21:39:18.133482 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:18.133459    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"]
Jul 02 21:39:18.137114 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:18.137092    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-e3b87-predictor-6c9ddc44bc-62kdb"]
Jul 02 21:39:19.013639 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:19.013606    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" path="/var/lib/kubelet/pods/a693b634-4747-43d7-8648-ff8d2843ad6d/volumes"
Jul 02 21:39:19.102518 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:19.102480    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9" event={"ID":"6394bdea-c72b-4f71-878f-10c98235628a","Type":"ContainerStarted","Data":"540f3adf9b7605927c3aa6defef57a83c9e3de952ab24d79d624e03f1a332f43"}
Jul 02 21:39:19.102646 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:19.102545    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9" event={"ID":"6394bdea-c72b-4f71-878f-10c98235628a","Type":"ContainerStarted","Data":"ab8719611d0abe69e41e909e377db4726693d4426cbf9ec4f755ed0d50a6162a"}
Jul 02 21:39:19.102756 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:19.102737    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:39:19.122226 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:19.122146    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9" podStartSLOduration=2.521131785 podStartE2EDuration="6.122128966s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.600997181s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:39:19.121816009 +0000 UTC m=+1230.748317908" watchObservedRunningTime="2026-07-02 21:39:19.122128966 +0000 UTC m=+1230.748630843"
Jul 02 21:39:20.107368 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:20.107339    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:39:20.108494 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:20.108469    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9" podUID="6394bdea-c72b-4f71-878f-10c98235628a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 02 21:39:21.110814 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:21.110769    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9" podUID="6394bdea-c72b-4f71-878f-10c98235628a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 02 21:39:26.115206 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:26.115159    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:39:26.117431 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:26.115764    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9" podUID="6394bdea-c72b-4f71-878f-10c98235628a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 02 21:39:36.116112 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:36.116075    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9" podUID="6394bdea-c72b-4f71-878f-10c98235628a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 02 21:39:46.115863 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:46.115775    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9" podUID="6394bdea-c72b-4f71-878f-10c98235628a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 02 21:39:56.116126 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:39:56.116081    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9" podUID="6394bdea-c72b-4f71-878f-10c98235628a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 02 21:40:06.115845 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:06.115802    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9" podUID="6394bdea-c72b-4f71-878f-10c98235628a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 02 21:40:16.116162 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:16.116119    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9" podUID="6394bdea-c72b-4f71-878f-10c98235628a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 02 21:40:26.116423 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:26.116385    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9" podUID="6394bdea-c72b-4f71-878f-10c98235628a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 02 21:40:28.010001 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:28.009974    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:40:33.233485 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.233451    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:40:33.267923 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.267898    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"]
Jul 02 21:40:33.268245 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.268233    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container"
Jul 02 21:40:33.268296 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.268248    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container"
Jul 02 21:40:33.268296 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.268263    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="storage-initializer"
Jul 02 21:40:33.268296 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.268269    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="storage-initializer"
Jul 02 21:40:33.268296 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.268284    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kube-rbac-proxy"
Jul 02 21:40:33.268296 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.268289    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kube-rbac-proxy"
Jul 02 21:40:33.268452 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.268338    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kube-rbac-proxy"
Jul 02 21:40:33.268452 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.268349    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a693b634-4747-43d7-8648-ff8d2843ad6d" containerName="kserve-container"
Jul 02 21:40:33.271504 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.271487    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"
Jul 02 21:40:33.274240 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.274223    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-0bbdb6-predictor-serving-cert\""
Jul 02 21:40:33.274452 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.274437    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-0bbdb6-kube-rbac-proxy-sar-config\""
Jul 02 21:40:33.274513 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.274460    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-0bbdb6-dockercfg-znnxs\""
Jul 02 21:40:33.274513 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.274437    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-0bbdb6\""
Jul 02 21:40:33.282308 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.282289    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"]
Jul 02 21:40:33.335335 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.335314    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b62245da-2522-464c-95ef-70d7d24c0122-kserve-provision-location\") pod \"isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr\" (UID: \"b62245da-2522-464c-95ef-70d7d24c0122\") " pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"
Jul 02 21:40:33.335431 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.335342    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p5xr\" (UniqueName: \"kubernetes.io/projected/b62245da-2522-464c-95ef-70d7d24c0122-kube-api-access-6p5xr\") pod \"isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr\" (UID: \"b62245da-2522-464c-95ef-70d7d24c0122\") " pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"
Jul 02 21:40:33.335431 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.335372    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b62245da-2522-464c-95ef-70d7d24c0122-proxy-tls\") pod \"isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr\" (UID: \"b62245da-2522-464c-95ef-70d7d24c0122\") " pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"
Jul 02 21:40:33.335431 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.335429    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b62245da-2522-464c-95ef-70d7d24c0122-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr\" (UID: \"b62245da-2522-464c-95ef-70d7d24c0122\") " pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"
Jul 02 21:40:33.335556 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.335470    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b62245da-2522-464c-95ef-70d7d24c0122-cabundle-cert\") pod \"isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr\" (UID: \"b62245da-2522-464c-95ef-70d7d24c0122\") " pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"
Jul 02 21:40:33.436195 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.436157    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b62245da-2522-464c-95ef-70d7d24c0122-kserve-provision-location\") pod \"isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr\" (UID: \"b62245da-2522-464c-95ef-70d7d24c0122\") " pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"
Jul 02 21:40:33.436317 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.436199    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p5xr\" (UniqueName: \"kubernetes.io/projected/b62245da-2522-464c-95ef-70d7d24c0122-kube-api-access-6p5xr\") pod \"isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr\" (UID: \"b62245da-2522-464c-95ef-70d7d24c0122\") " pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"
Jul 02 21:40:33.436317 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.436228    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b62245da-2522-464c-95ef-70d7d24c0122-proxy-tls\") pod \"isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr\" (UID: \"b62245da-2522-464c-95ef-70d7d24c0122\") " pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"
Jul 02 21:40:33.436432 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.436332    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b62245da-2522-464c-95ef-70d7d24c0122-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr\" (UID: \"b62245da-2522-464c-95ef-70d7d24c0122\") " pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"
Jul 02 21:40:33.436432 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.436413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b62245da-2522-464c-95ef-70d7d24c0122-cabundle-cert\") pod \"isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr\" (UID: \"b62245da-2522-464c-95ef-70d7d24c0122\") " pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"
Jul 02 21:40:33.436598 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.436580    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b62245da-2522-464c-95ef-70d7d24c0122-kserve-provision-location\") pod \"isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr\" (UID: \"b62245da-2522-464c-95ef-70d7d24c0122\") " pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"
Jul 02 21:40:33.437010 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.436989    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b62245da-2522-464c-95ef-70d7d24c0122-cabundle-cert\") pod \"isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr\" (UID: \"b62245da-2522-464c-95ef-70d7d24c0122\") " pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"
Jul 02 21:40:33.437010 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.437000    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b62245da-2522-464c-95ef-70d7d24c0122-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr\" (UID: \"b62245da-2522-464c-95ef-70d7d24c0122\") " pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"
Jul 02 21:40:33.438602 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.438583    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b62245da-2522-464c-95ef-70d7d24c0122-proxy-tls\") pod \"isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr\" (UID: \"b62245da-2522-464c-95ef-70d7d24c0122\") " pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"
Jul 02 21:40:33.444385 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.444352    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p5xr\" (UniqueName: \"kubernetes.io/projected/b62245da-2522-464c-95ef-70d7d24c0122-kube-api-access-6p5xr\") pod \"isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr\" (UID: \"b62245da-2522-464c-95ef-70d7d24c0122\") " pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"
Jul 02 21:40:33.582415 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.582354    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"
Jul 02 21:40:33.709371 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:33.709348    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"]
Jul 02 21:40:33.711446 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:40:33.711420    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb62245da_2522_464c_95ef_70d7d24c0122.slice/crio-87a9a7e0a348b84d80aa31ead65eea1c2863c814f6782d7ce8ef98cafb3214a7 WatchSource:0}: Error finding container 87a9a7e0a348b84d80aa31ead65eea1c2863c814f6782d7ce8ef98cafb3214a7: Status 404 returned error can't find the container with id 87a9a7e0a348b84d80aa31ead65eea1c2863c814f6782d7ce8ef98cafb3214a7
Jul 02 21:40:34.402726 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:34.402684    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr" event={"ID":"b62245da-2522-464c-95ef-70d7d24c0122","Type":"ContainerStarted","Data":"6412d0922e9a26e70c444cba03c0234ed69cd7c52725603b422704a325d6847e"}
Jul 02 21:40:34.403230 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:34.402757    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr" event={"ID":"b62245da-2522-464c-95ef-70d7d24c0122","Type":"ContainerStarted","Data":"87a9a7e0a348b84d80aa31ead65eea1c2863c814f6782d7ce8ef98cafb3214a7"}
Jul 02 21:40:34.478285 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:34.478248    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:40:38.419312 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:38.419286    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr_b62245da-2522-464c-95ef-70d7d24c0122/storage-initializer/0.log"
Jul 02 21:40:38.419699 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:38.419323    2575 generic.go:356] "Generic (PLEG): container finished" podID="b62245da-2522-464c-95ef-70d7d24c0122" containerID="6412d0922e9a26e70c444cba03c0234ed69cd7c52725603b422704a325d6847e" exitCode=1
Jul 02 21:40:38.419699 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:38.419403    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr" event={"ID":"b62245da-2522-464c-95ef-70d7d24c0122","Type":"ContainerDied","Data":"6412d0922e9a26e70c444cba03c0234ed69cd7c52725603b422704a325d6847e"}
Jul 02 21:40:39.425045 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:39.425019    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr_b62245da-2522-464c-95ef-70d7d24c0122/storage-initializer/0.log"
Jul 02 21:40:39.425483 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:39.425069    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr" event={"ID":"b62245da-2522-464c-95ef-70d7d24c0122","Type":"ContainerStarted","Data":"4b3ac6dd29f66d051be841b634f99deeb1eb23c3ee97bae26b6b06ad576f17d7"}
Jul 02 21:40:42.440701 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:42.440676    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr_b62245da-2522-464c-95ef-70d7d24c0122/storage-initializer/1.log"
Jul 02 21:40:42.441121 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:42.441100    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr_b62245da-2522-464c-95ef-70d7d24c0122/storage-initializer/0.log"
Jul 02 21:40:42.441213 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:42.441139    2575 generic.go:356] "Generic (PLEG): container finished" podID="b62245da-2522-464c-95ef-70d7d24c0122" containerID="4b3ac6dd29f66d051be841b634f99deeb1eb23c3ee97bae26b6b06ad576f17d7" exitCode=1
Jul 02 21:40:42.441213 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:42.441199    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr" event={"ID":"b62245da-2522-464c-95ef-70d7d24c0122","Type":"ContainerDied","Data":"4b3ac6dd29f66d051be841b634f99deeb1eb23c3ee97bae26b6b06ad576f17d7"}
Jul 02 21:40:42.441317 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:42.441261    2575 scope.go:117] "RemoveContainer" containerID="6412d0922e9a26e70c444cba03c0234ed69cd7c52725603b422704a325d6847e"
Jul 02 21:40:42.441630 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:42.441613    2575 scope.go:117] "RemoveContainer" containerID="6412d0922e9a26e70c444cba03c0234ed69cd7c52725603b422704a325d6847e"
Jul 02 21:40:42.451716 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:40:42.451693    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr_kserve-ci-e2e-test(b62245da-2522-464c-95ef-70d7d24c0122)\"" pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr" podUID="b62245da-2522-464c-95ef-70d7d24c0122"
Jul 02 21:40:42.451881 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:40:42.451854    2575 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr_kserve-ci-e2e-test_b62245da-2522-464c-95ef-70d7d24c0122_0 in pod sandbox 87a9a7e0a348b84d80aa31ead65eea1c2863c814f6782d7ce8ef98cafb3214a7 from index: no such id: '6412d0922e9a26e70c444cba03c0234ed69cd7c52725603b422704a325d6847e'" containerID="6412d0922e9a26e70c444cba03c0234ed69cd7c52725603b422704a325d6847e"
Jul 02 21:40:42.451943 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:42.451899    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6412d0922e9a26e70c444cba03c0234ed69cd7c52725603b422704a325d6847e"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr_kserve-ci-e2e-test_b62245da-2522-464c-95ef-70d7d24c0122_0 in pod sandbox 87a9a7e0a348b84d80aa31ead65eea1c2863c814f6782d7ce8ef98cafb3214a7 from index: no such id: '6412d0922e9a26e70c444cba03c0234ed69cd7c52725603b422704a325d6847e'"
Jul 02 21:40:43.447208 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:43.447155    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr_b62245da-2522-464c-95ef-70d7d24c0122/storage-initializer/1.log"
Jul 02 21:40:49.332674 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.332636    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"]
Jul 02 21:40:49.356573 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.356513    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:40:49.392141 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.392112    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"]
Jul 02 21:40:49.392402 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.392374    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9" podUID="6394bdea-c72b-4f71-878f-10c98235628a" containerName="kserve-container" containerID="cri-o://ab8719611d0abe69e41e909e377db4726693d4426cbf9ec4f755ed0d50a6162a" gracePeriod=30
Jul 02 21:40:49.392503 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.392406    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9" podUID="6394bdea-c72b-4f71-878f-10c98235628a" containerName="kube-rbac-proxy" containerID="cri-o://540f3adf9b7605927c3aa6defef57a83c9e3de952ab24d79d624e03f1a332f43" gracePeriod=30
Jul 02 21:40:49.420902 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.420877    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:40:49.512007 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.511989    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr_b62245da-2522-464c-95ef-70d7d24c0122/storage-initializer/1.log"
Jul 02 21:40:49.514019 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.514005    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"
Jul 02 21:40:49.673433 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.673402    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b62245da-2522-464c-95ef-70d7d24c0122-kserve-provision-location\") pod \"b62245da-2522-464c-95ef-70d7d24c0122\" (UID: \"b62245da-2522-464c-95ef-70d7d24c0122\") "
Jul 02 21:40:49.673596 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.673444    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b62245da-2522-464c-95ef-70d7d24c0122-proxy-tls\") pod \"b62245da-2522-464c-95ef-70d7d24c0122\" (UID: \"b62245da-2522-464c-95ef-70d7d24c0122\") "
Jul 02 21:40:49.673596 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.673473    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6p5xr\" (UniqueName: \"kubernetes.io/projected/b62245da-2522-464c-95ef-70d7d24c0122-kube-api-access-6p5xr\") pod \"b62245da-2522-464c-95ef-70d7d24c0122\" (UID: \"b62245da-2522-464c-95ef-70d7d24c0122\") "
Jul 02 21:40:49.673596 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.673523    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b62245da-2522-464c-95ef-70d7d24c0122-kube-rbac-proxy-sar-config\") pod \"b62245da-2522-464c-95ef-70d7d24c0122\" (UID: \"b62245da-2522-464c-95ef-70d7d24c0122\") "
Jul 02 21:40:49.673596 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.673558    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b62245da-2522-464c-95ef-70d7d24c0122-cabundle-cert\") pod \"b62245da-2522-464c-95ef-70d7d24c0122\" (UID: \"b62245da-2522-464c-95ef-70d7d24c0122\") "
Jul 02 21:40:49.673766 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.673720    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b62245da-2522-464c-95ef-70d7d24c0122-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b62245da-2522-464c-95ef-70d7d24c0122" (UID: "b62245da-2522-464c-95ef-70d7d24c0122"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:40:49.673926 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.673897    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b62245da-2522-464c-95ef-70d7d24c0122-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b62245da-2522-464c-95ef-70d7d24c0122" (UID: "b62245da-2522-464c-95ef-70d7d24c0122"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:40:49.673926 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.673912    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b62245da-2522-464c-95ef-70d7d24c0122-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b62245da-2522-464c-95ef-70d7d24c0122" (UID: "b62245da-2522-464c-95ef-70d7d24c0122"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:40:49.675575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.675556    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b62245da-2522-464c-95ef-70d7d24c0122-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b62245da-2522-464c-95ef-70d7d24c0122" (UID: "b62245da-2522-464c-95ef-70d7d24c0122"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:40:49.675649 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.675576    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b62245da-2522-464c-95ef-70d7d24c0122-kube-api-access-6p5xr" (OuterVolumeSpecName: "kube-api-access-6p5xr") pod "b62245da-2522-464c-95ef-70d7d24c0122" (UID: "b62245da-2522-464c-95ef-70d7d24c0122"). InnerVolumeSpecName "kube-api-access-6p5xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:40:49.774335 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.774311    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b62245da-2522-464c-95ef-70d7d24c0122-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:40:49.774449 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.774351    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b62245da-2522-464c-95ef-70d7d24c0122-cabundle-cert\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:40:49.774449 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.774361    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b62245da-2522-464c-95ef-70d7d24c0122-kserve-provision-location\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:40:49.774449 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.774370    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b62245da-2522-464c-95ef-70d7d24c0122-proxy-tls\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:40:49.774449 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:49.774378    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p5xr\" (UniqueName: \"kubernetes.io/projected/b62245da-2522-464c-95ef-70d7d24c0122-kube-api-access-6p5xr\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:40:50.476391 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:50.476365    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr_b62245da-2522-464c-95ef-70d7d24c0122/storage-initializer/1.log"
Jul 02 21:40:50.476787 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:50.476466    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"
Jul 02 21:40:50.476787 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:50.476474    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr" event={"ID":"b62245da-2522-464c-95ef-70d7d24c0122","Type":"ContainerDied","Data":"87a9a7e0a348b84d80aa31ead65eea1c2863c814f6782d7ce8ef98cafb3214a7"}
Jul 02 21:40:50.476787 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:50.476545    2575 scope.go:117] "RemoveContainer" containerID="4b3ac6dd29f66d051be841b634f99deeb1eb23c3ee97bae26b6b06ad576f17d7"
Jul 02 21:40:50.478777 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:50.478753    2575 generic.go:356] "Generic (PLEG): container finished" podID="6394bdea-c72b-4f71-878f-10c98235628a" containerID="540f3adf9b7605927c3aa6defef57a83c9e3de952ab24d79d624e03f1a332f43" exitCode=2
Jul 02 21:40:50.478907 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:50.478803    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9" event={"ID":"6394bdea-c72b-4f71-878f-10c98235628a","Type":"ContainerDied","Data":"540f3adf9b7605927c3aa6defef57a83c9e3de952ab24d79d624e03f1a332f43"}
Jul 02 21:40:50.518449 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:50.518424    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"]
Jul 02 21:40:50.522221 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:50.522201    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0bbdb6-predictor-7d847756c9-n48dr"]
Jul 02 21:40:51.014066 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:51.014031    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b62245da-2522-464c-95ef-70d7d24c0122" path="/var/lib/kubelet/pods/b62245da-2522-464c-95ef-70d7d24c0122/volumes"
Jul 02 21:40:51.111034 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:51.110990    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9" podUID="6394bdea-c72b-4f71-878f-10c98235628a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Jul 02 21:40:53.494203 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:53.494151    2575 generic.go:356] "Generic (PLEG): container finished" podID="6394bdea-c72b-4f71-878f-10c98235628a" containerID="ab8719611d0abe69e41e909e377db4726693d4426cbf9ec4f755ed0d50a6162a" exitCode=0
Jul 02 21:40:53.494523 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:53.494206    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9" event={"ID":"6394bdea-c72b-4f71-878f-10c98235628a","Type":"ContainerDied","Data":"ab8719611d0abe69e41e909e377db4726693d4426cbf9ec4f755ed0d50a6162a"}
Jul 02 21:40:53.529740 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:53.529719    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:40:53.706709 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:53.706682    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6394bdea-c72b-4f71-878f-10c98235628a-kserve-provision-location\") pod \"6394bdea-c72b-4f71-878f-10c98235628a\" (UID: \"6394bdea-c72b-4f71-878f-10c98235628a\") "
Jul 02 21:40:53.706867 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:53.706721    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6394bdea-c72b-4f71-878f-10c98235628a-proxy-tls\") pod \"6394bdea-c72b-4f71-878f-10c98235628a\" (UID: \"6394bdea-c72b-4f71-878f-10c98235628a\") "
Jul 02 21:40:53.706867 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:53.706761    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rr46\" (UniqueName: \"kubernetes.io/projected/6394bdea-c72b-4f71-878f-10c98235628a-kube-api-access-7rr46\") pod \"6394bdea-c72b-4f71-878f-10c98235628a\" (UID: \"6394bdea-c72b-4f71-878f-10c98235628a\") "
Jul 02 21:40:53.706867 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:53.706786    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6394bdea-c72b-4f71-878f-10c98235628a-cabundle-cert\") pod \"6394bdea-c72b-4f71-878f-10c98235628a\" (UID: \"6394bdea-c72b-4f71-878f-10c98235628a\") "
Jul 02 21:40:53.706867 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:53.706806    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6394bdea-c72b-4f71-878f-10c98235628a-kube-rbac-proxy-sar-config\") pod \"6394bdea-c72b-4f71-878f-10c98235628a\" (UID: \"6394bdea-c72b-4f71-878f-10c98235628a\") "
Jul 02 21:40:53.707094 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:53.707001    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6394bdea-c72b-4f71-878f-10c98235628a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6394bdea-c72b-4f71-878f-10c98235628a" (UID: "6394bdea-c72b-4f71-878f-10c98235628a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:40:53.707203 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:53.707152    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6394bdea-c72b-4f71-878f-10c98235628a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6394bdea-c72b-4f71-878f-10c98235628a" (UID: "6394bdea-c72b-4f71-878f-10c98235628a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:40:53.707253 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:53.707228    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6394bdea-c72b-4f71-878f-10c98235628a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6394bdea-c72b-4f71-878f-10c98235628a" (UID: "6394bdea-c72b-4f71-878f-10c98235628a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:40:53.708881 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:53.708849    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6394bdea-c72b-4f71-878f-10c98235628a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6394bdea-c72b-4f71-878f-10c98235628a" (UID: "6394bdea-c72b-4f71-878f-10c98235628a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:40:53.708987 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:53.708922    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6394bdea-c72b-4f71-878f-10c98235628a-kube-api-access-7rr46" (OuterVolumeSpecName: "kube-api-access-7rr46") pod "6394bdea-c72b-4f71-878f-10c98235628a" (UID: "6394bdea-c72b-4f71-878f-10c98235628a"). InnerVolumeSpecName "kube-api-access-7rr46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:40:53.807501 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:53.807478    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6394bdea-c72b-4f71-878f-10c98235628a-kserve-provision-location\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:40:53.807501 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:53.807498    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6394bdea-c72b-4f71-878f-10c98235628a-proxy-tls\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:40:53.807635 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:53.807507    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rr46\" (UniqueName: \"kubernetes.io/projected/6394bdea-c72b-4f71-878f-10c98235628a-kube-api-access-7rr46\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:40:53.807635 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:53.807516    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6394bdea-c72b-4f71-878f-10c98235628a-cabundle-cert\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:40:53.807635 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:53.807524    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6394bdea-c72b-4f71-878f-10c98235628a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:40:54.500188 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.500143    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9" event={"ID":"6394bdea-c72b-4f71-878f-10c98235628a","Type":"ContainerDied","Data":"20c9645279014e42799c56c78a83a6bac09f405f148320d18028bb5c69549f21"}
Jul 02 21:40:54.500595 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.500219    2575 scope.go:117] "RemoveContainer" containerID="540f3adf9b7605927c3aa6defef57a83c9e3de952ab24d79d624e03f1a332f43"
Jul 02 21:40:54.500595 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.500158    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"
Jul 02 21:40:54.508320 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.508303    2575 scope.go:117] "RemoveContainer" containerID="ab8719611d0abe69e41e909e377db4726693d4426cbf9ec4f755ed0d50a6162a"
Jul 02 21:40:54.513104 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.513080    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:40:54.516342 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.516320    2575 scope.go:117] "RemoveContainer" containerID="edc27d0a6d1e3a061cf0b14971616dc52ceeb438e0889b8dde0e18e4fab3d524"
Jul 02 21:40:54.525924 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.525903    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"]
Jul 02 21:40:54.527260 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.527236    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0bbdb6-predictor-847fc894cb-g2zr9"]
Jul 02 21:40:54.547093 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.547068    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"]
Jul 02 21:40:54.547375 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.547363    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6394bdea-c72b-4f71-878f-10c98235628a" containerName="kserve-container"
Jul 02 21:40:54.547465 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.547377    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6394bdea-c72b-4f71-878f-10c98235628a" containerName="kserve-container"
Jul 02 21:40:54.547465 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.547387    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6394bdea-c72b-4f71-878f-10c98235628a" containerName="storage-initializer"
Jul 02 21:40:54.547465 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.547393    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6394bdea-c72b-4f71-878f-10c98235628a" containerName="storage-initializer"
Jul 02 21:40:54.547465 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.547399    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b62245da-2522-464c-95ef-70d7d24c0122" containerName="storage-initializer"
Jul 02 21:40:54.547465 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.547404    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b62245da-2522-464c-95ef-70d7d24c0122" containerName="storage-initializer"
Jul 02 21:40:54.547465 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.547413    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b62245da-2522-464c-95ef-70d7d24c0122" containerName="storage-initializer"
Jul 02 21:40:54.547465 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.547418    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b62245da-2522-464c-95ef-70d7d24c0122" containerName="storage-initializer"
Jul 02 21:40:54.547465 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.547438    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6394bdea-c72b-4f71-878f-10c98235628a" containerName="kube-rbac-proxy"
Jul 02 21:40:54.547465 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.547444    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6394bdea-c72b-4f71-878f-10c98235628a" containerName="kube-rbac-proxy"
Jul 02 21:40:54.547717 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.547491    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6394bdea-c72b-4f71-878f-10c98235628a" containerName="kserve-container"
Jul 02 21:40:54.547717 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.547501    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6394bdea-c72b-4f71-878f-10c98235628a" containerName="kube-rbac-proxy"
Jul 02 21:40:54.547717 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.547508    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b62245da-2522-464c-95ef-70d7d24c0122" containerName="storage-initializer"
Jul 02 21:40:54.547717 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.547516    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b62245da-2522-464c-95ef-70d7d24c0122" containerName="storage-initializer"
Jul 02 21:40:54.551690 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.551673    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"
Jul 02 21:40:54.554867 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.554849    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-02b5ab-predictor-serving-cert\""
Jul 02 21:40:54.554967 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.554934    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 21:40:54.555210 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.555189    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-02b5ab-kube-rbac-proxy-sar-config\""
Jul 02 21:40:54.555334 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.555312    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 21:40:54.555334 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.555326    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 21:40:54.555484 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.555472    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-02b5ab\""
Jul 02 21:40:54.555583 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.555567    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-02b5ab-dockercfg-82bt6\""
Jul 02 21:40:54.567037 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.567017    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"]
Jul 02 21:40:54.708449 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.708419    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:40:54.714508 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.714488    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be581cf6-68df-41db-8a1a-e8a269a742fe-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-02b5ab-predictor-594949977c-bw5rb\" (UID: \"be581cf6-68df-41db-8a1a-e8a269a742fe\") " pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"
Jul 02 21:40:54.714614 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.714543    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be581cf6-68df-41db-8a1a-e8a269a742fe-proxy-tls\") pod \"isvc-init-fail-02b5ab-predictor-594949977c-bw5rb\" (UID: \"be581cf6-68df-41db-8a1a-e8a269a742fe\") " pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"
Jul 02 21:40:54.714674 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.714624    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be581cf6-68df-41db-8a1a-e8a269a742fe-kserve-provision-location\") pod \"isvc-init-fail-02b5ab-predictor-594949977c-bw5rb\" (UID: \"be581cf6-68df-41db-8a1a-e8a269a742fe\") " pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"
Jul 02 21:40:54.714674 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.714657    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxqp8\" (UniqueName: \"kubernetes.io/projected/be581cf6-68df-41db-8a1a-e8a269a742fe-kube-api-access-zxqp8\") pod \"isvc-init-fail-02b5ab-predictor-594949977c-bw5rb\" (UID: \"be581cf6-68df-41db-8a1a-e8a269a742fe\") " pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"
Jul 02 21:40:54.714755 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.714675    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be581cf6-68df-41db-8a1a-e8a269a742fe-cabundle-cert\") pod \"isvc-init-fail-02b5ab-predictor-594949977c-bw5rb\" (UID: \"be581cf6-68df-41db-8a1a-e8a269a742fe\") " pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"
Jul 02 21:40:54.815848 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.815775    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be581cf6-68df-41db-8a1a-e8a269a742fe-kserve-provision-location\") pod \"isvc-init-fail-02b5ab-predictor-594949977c-bw5rb\" (UID: \"be581cf6-68df-41db-8a1a-e8a269a742fe\") " pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"
Jul 02 21:40:54.815848 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.815815    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxqp8\" (UniqueName: \"kubernetes.io/projected/be581cf6-68df-41db-8a1a-e8a269a742fe-kube-api-access-zxqp8\") pod \"isvc-init-fail-02b5ab-predictor-594949977c-bw5rb\" (UID: \"be581cf6-68df-41db-8a1a-e8a269a742fe\") " pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"
Jul 02 21:40:54.815848 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.815833    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be581cf6-68df-41db-8a1a-e8a269a742fe-cabundle-cert\") pod \"isvc-init-fail-02b5ab-predictor-594949977c-bw5rb\" (UID: \"be581cf6-68df-41db-8a1a-e8a269a742fe\") " pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"
Jul 02 21:40:54.816066 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.815858    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be581cf6-68df-41db-8a1a-e8a269a742fe-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-02b5ab-predictor-594949977c-bw5rb\" (UID: \"be581cf6-68df-41db-8a1a-e8a269a742fe\") " pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"
Jul 02 21:40:54.816066 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.815930    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be581cf6-68df-41db-8a1a-e8a269a742fe-proxy-tls\") pod \"isvc-init-fail-02b5ab-predictor-594949977c-bw5rb\" (UID: \"be581cf6-68df-41db-8a1a-e8a269a742fe\") " pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"
Jul 02 21:40:54.816256 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.816238    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be581cf6-68df-41db-8a1a-e8a269a742fe-kserve-provision-location\") pod \"isvc-init-fail-02b5ab-predictor-594949977c-bw5rb\" (UID: \"be581cf6-68df-41db-8a1a-e8a269a742fe\") " pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"
Jul 02 21:40:54.816612 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.816587    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be581cf6-68df-41db-8a1a-e8a269a742fe-cabundle-cert\") pod \"isvc-init-fail-02b5ab-predictor-594949977c-bw5rb\" (UID: \"be581cf6-68df-41db-8a1a-e8a269a742fe\") " pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"
Jul 02 21:40:54.816694 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.816587    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be581cf6-68df-41db-8a1a-e8a269a742fe-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-02b5ab-predictor-594949977c-bw5rb\" (UID: \"be581cf6-68df-41db-8a1a-e8a269a742fe\") " pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"
Jul 02 21:40:54.818190 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.818152    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be581cf6-68df-41db-8a1a-e8a269a742fe-proxy-tls\") pod \"isvc-init-fail-02b5ab-predictor-594949977c-bw5rb\" (UID: \"be581cf6-68df-41db-8a1a-e8a269a742fe\") " pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"
Jul 02 21:40:54.826610 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.826572    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxqp8\" (UniqueName: \"kubernetes.io/projected/be581cf6-68df-41db-8a1a-e8a269a742fe-kube-api-access-zxqp8\") pod \"isvc-init-fail-02b5ab-predictor-594949977c-bw5rb\" (UID: \"be581cf6-68df-41db-8a1a-e8a269a742fe\") " pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"
Jul 02 21:40:54.861632 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.861606    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"
Jul 02 21:40:54.990346 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:54.990325    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"]
Jul 02 21:40:54.992721 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:40:54.992691    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe581cf6_68df_41db_8a1a_e8a269a742fe.slice/crio-a41f3850dbf9436f2144dc049cc67d80e102561de3da5187f6b0af630d573bf6 WatchSource:0}: Error finding container a41f3850dbf9436f2144dc049cc67d80e102561de3da5187f6b0af630d573bf6: Status 404 returned error can't find the container with id a41f3850dbf9436f2144dc049cc67d80e102561de3da5187f6b0af630d573bf6
Jul 02 21:40:55.013844 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:55.013818    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6394bdea-c72b-4f71-878f-10c98235628a" path="/var/lib/kubelet/pods/6394bdea-c72b-4f71-878f-10c98235628a/volumes"
Jul 02 21:40:55.506474 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:55.506441    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb" event={"ID":"be581cf6-68df-41db-8a1a-e8a269a742fe","Type":"ContainerStarted","Data":"d1e4aa1be597f860a3bc0f65b9e6ad4a60701319a3533fb7605a55f7f4344554"}
Jul 02 21:40:55.506858 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:55.506503    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb" event={"ID":"be581cf6-68df-41db-8a1a-e8a269a742fe","Type":"ContainerStarted","Data":"a41f3850dbf9436f2144dc049cc67d80e102561de3da5187f6b0af630d573bf6"}
Jul 02 21:40:56.555927 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:56.555895    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:40:58.519273 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:58.519243    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-02b5ab-predictor-594949977c-bw5rb_be581cf6-68df-41db-8a1a-e8a269a742fe/storage-initializer/0.log"
Jul 02 21:40:58.519643 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:58.519286    2575 generic.go:356] "Generic (PLEG): container finished" podID="be581cf6-68df-41db-8a1a-e8a269a742fe" containerID="d1e4aa1be597f860a3bc0f65b9e6ad4a60701319a3533fb7605a55f7f4344554" exitCode=1
Jul 02 21:40:58.519643 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:58.519368    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb" event={"ID":"be581cf6-68df-41db-8a1a-e8a269a742fe","Type":"ContainerDied","Data":"d1e4aa1be597f860a3bc0f65b9e6ad4a60701319a3533fb7605a55f7f4344554"}
Jul 02 21:40:59.520822 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:59.520794    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"]
Jul 02 21:40:59.526360 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:59.526335    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-02b5ab-predictor-594949977c-bw5rb_be581cf6-68df-41db-8a1a-e8a269a742fe/storage-initializer/0.log"
Jul 02 21:40:59.526505 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:59.526421    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb" event={"ID":"be581cf6-68df-41db-8a1a-e8a269a742fe","Type":"ContainerStarted","Data":"57c6691313158efbe925bc7ef33f5e81f26097336b15dee8e5fcbff5821ae7b6"}
Jul 02 21:40:59.526569 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:59.526496    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb" podUID="be581cf6-68df-41db-8a1a-e8a269a742fe" containerName="storage-initializer" containerID="cri-o://57c6691313158efbe925bc7ef33f5e81f26097336b15dee8e5fcbff5821ae7b6" gracePeriod=30
Jul 02 21:40:59.547806 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:40:59.547774    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:41:01.472080 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.472057    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-02b5ab-predictor-594949977c-bw5rb_be581cf6-68df-41db-8a1a-e8a269a742fe/storage-initializer/1.log"
Jul 02 21:41:01.472497 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.472481    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-02b5ab-predictor-594949977c-bw5rb_be581cf6-68df-41db-8a1a-e8a269a742fe/storage-initializer/0.log"
Jul 02 21:41:01.474678 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.474663    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"
Jul 02 21:41:01.534915 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.534891    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-02b5ab-predictor-594949977c-bw5rb_be581cf6-68df-41db-8a1a-e8a269a742fe/storage-initializer/1.log"
Jul 02 21:41:01.535356 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.535340    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-02b5ab-predictor-594949977c-bw5rb_be581cf6-68df-41db-8a1a-e8a269a742fe/storage-initializer/0.log"
Jul 02 21:41:01.535420 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.535375    2575 generic.go:356] "Generic (PLEG): container finished" podID="be581cf6-68df-41db-8a1a-e8a269a742fe" containerID="57c6691313158efbe925bc7ef33f5e81f26097336b15dee8e5fcbff5821ae7b6" exitCode=1
Jul 02 21:41:01.535454 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.535430    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"
Jul 02 21:41:01.535454 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.535441    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb" event={"ID":"be581cf6-68df-41db-8a1a-e8a269a742fe","Type":"ContainerDied","Data":"57c6691313158efbe925bc7ef33f5e81f26097336b15dee8e5fcbff5821ae7b6"}
Jul 02 21:41:01.535515 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.535495    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb" event={"ID":"be581cf6-68df-41db-8a1a-e8a269a742fe","Type":"ContainerDied","Data":"a41f3850dbf9436f2144dc049cc67d80e102561de3da5187f6b0af630d573bf6"}
Jul 02 21:41:01.535551 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.535523    2575 scope.go:117] "RemoveContainer" containerID="57c6691313158efbe925bc7ef33f5e81f26097336b15dee8e5fcbff5821ae7b6"
Jul 02 21:41:01.544479 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.544459    2575 scope.go:117] "RemoveContainer" containerID="d1e4aa1be597f860a3bc0f65b9e6ad4a60701319a3533fb7605a55f7f4344554"
Jul 02 21:41:01.551874 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.551842    2575 scope.go:117] "RemoveContainer" containerID="57c6691313158efbe925bc7ef33f5e81f26097336b15dee8e5fcbff5821ae7b6"
Jul 02 21:41:01.552121 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:41:01.552100    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57c6691313158efbe925bc7ef33f5e81f26097336b15dee8e5fcbff5821ae7b6\": container with ID starting with 57c6691313158efbe925bc7ef33f5e81f26097336b15dee8e5fcbff5821ae7b6 not found: ID does not exist" containerID="57c6691313158efbe925bc7ef33f5e81f26097336b15dee8e5fcbff5821ae7b6"
Jul 02 21:41:01.552198 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.552129    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c6691313158efbe925bc7ef33f5e81f26097336b15dee8e5fcbff5821ae7b6"} err="failed to get container status \"57c6691313158efbe925bc7ef33f5e81f26097336b15dee8e5fcbff5821ae7b6\": rpc error: code = NotFound desc = could not find container \"57c6691313158efbe925bc7ef33f5e81f26097336b15dee8e5fcbff5821ae7b6\": container with ID starting with 57c6691313158efbe925bc7ef33f5e81f26097336b15dee8e5fcbff5821ae7b6 not found: ID does not exist"
Jul 02 21:41:01.552198 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.552145    2575 scope.go:117] "RemoveContainer" containerID="d1e4aa1be597f860a3bc0f65b9e6ad4a60701319a3533fb7605a55f7f4344554"
Jul 02 21:41:01.552391 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:41:01.552374    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1e4aa1be597f860a3bc0f65b9e6ad4a60701319a3533fb7605a55f7f4344554\": container with ID starting with d1e4aa1be597f860a3bc0f65b9e6ad4a60701319a3533fb7605a55f7f4344554 not found: ID does not exist" containerID="d1e4aa1be597f860a3bc0f65b9e6ad4a60701319a3533fb7605a55f7f4344554"
Jul 02 21:41:01.552435 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.552396    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1e4aa1be597f860a3bc0f65b9e6ad4a60701319a3533fb7605a55f7f4344554"} err="failed to get container status \"d1e4aa1be597f860a3bc0f65b9e6ad4a60701319a3533fb7605a55f7f4344554\": rpc error: code = NotFound desc = could not find container \"d1e4aa1be597f860a3bc0f65b9e6ad4a60701319a3533fb7605a55f7f4344554\": container with ID starting with d1e4aa1be597f860a3bc0f65b9e6ad4a60701319a3533fb7605a55f7f4344554 not found: ID does not exist"
Jul 02 21:41:01.571688 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.571660    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be581cf6-68df-41db-8a1a-e8a269a742fe-kube-rbac-proxy-sar-config\") pod \"be581cf6-68df-41db-8a1a-e8a269a742fe\" (UID: \"be581cf6-68df-41db-8a1a-e8a269a742fe\") "
Jul 02 21:41:01.571779 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.571728    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxqp8\" (UniqueName: \"kubernetes.io/projected/be581cf6-68df-41db-8a1a-e8a269a742fe-kube-api-access-zxqp8\") pod \"be581cf6-68df-41db-8a1a-e8a269a742fe\" (UID: \"be581cf6-68df-41db-8a1a-e8a269a742fe\") "
Jul 02 21:41:01.571779 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.571759    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be581cf6-68df-41db-8a1a-e8a269a742fe-proxy-tls\") pod \"be581cf6-68df-41db-8a1a-e8a269a742fe\" (UID: \"be581cf6-68df-41db-8a1a-e8a269a742fe\") "
Jul 02 21:41:01.571856 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.571812    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be581cf6-68df-41db-8a1a-e8a269a742fe-kserve-provision-location\") pod \"be581cf6-68df-41db-8a1a-e8a269a742fe\" (UID: \"be581cf6-68df-41db-8a1a-e8a269a742fe\") "
Jul 02 21:41:01.571915 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.571907    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be581cf6-68df-41db-8a1a-e8a269a742fe-cabundle-cert\") pod \"be581cf6-68df-41db-8a1a-e8a269a742fe\" (UID: \"be581cf6-68df-41db-8a1a-e8a269a742fe\") "
Jul 02 21:41:01.572057 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.572033    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be581cf6-68df-41db-8a1a-e8a269a742fe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "be581cf6-68df-41db-8a1a-e8a269a742fe" (UID: "be581cf6-68df-41db-8a1a-e8a269a742fe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:41:01.572183 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.572110    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be581cf6-68df-41db-8a1a-e8a269a742fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "be581cf6-68df-41db-8a1a-e8a269a742fe" (UID: "be581cf6-68df-41db-8a1a-e8a269a742fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:41:01.572324 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.572303    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be581cf6-68df-41db-8a1a-e8a269a742fe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "be581cf6-68df-41db-8a1a-e8a269a742fe" (UID: "be581cf6-68df-41db-8a1a-e8a269a742fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:41:01.573842 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.573823    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be581cf6-68df-41db-8a1a-e8a269a742fe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "be581cf6-68df-41db-8a1a-e8a269a742fe" (UID: "be581cf6-68df-41db-8a1a-e8a269a742fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:41:01.573931 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.573860    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be581cf6-68df-41db-8a1a-e8a269a742fe-kube-api-access-zxqp8" (OuterVolumeSpecName: "kube-api-access-zxqp8") pod "be581cf6-68df-41db-8a1a-e8a269a742fe" (UID: "be581cf6-68df-41db-8a1a-e8a269a742fe"). InnerVolumeSpecName "kube-api-access-zxqp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:41:01.673031 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.672995    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be581cf6-68df-41db-8a1a-e8a269a742fe-cabundle-cert\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:41:01.673031 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.673025    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be581cf6-68df-41db-8a1a-e8a269a742fe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:41:01.673031 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.673035    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxqp8\" (UniqueName: \"kubernetes.io/projected/be581cf6-68df-41db-8a1a-e8a269a742fe-kube-api-access-zxqp8\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:41:01.673285 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.673045    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be581cf6-68df-41db-8a1a-e8a269a742fe-proxy-tls\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:41:01.673285 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.673054    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be581cf6-68df-41db-8a1a-e8a269a742fe-kserve-provision-location\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:41:01.873725 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.873696    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"]
Jul 02 21:41:01.880228 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:01.880202    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-02b5ab-predictor-594949977c-bw5rb"]
Jul 02 21:41:03.013743 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:03.013710    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be581cf6-68df-41db-8a1a-e8a269a742fe" path="/var/lib/kubelet/pods/be581cf6-68df-41db-8a1a-e8a269a742fe/volumes"
Jul 02 21:41:04.713998 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.713971    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:41:04.739834 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.739806    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"]
Jul 02 21:41:04.740150 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.740138    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be581cf6-68df-41db-8a1a-e8a269a742fe" containerName="storage-initializer"
Jul 02 21:41:04.740150 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.740152    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="be581cf6-68df-41db-8a1a-e8a269a742fe" containerName="storage-initializer"
Jul 02 21:41:04.740303 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.740242    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="be581cf6-68df-41db-8a1a-e8a269a742fe" containerName="storage-initializer"
Jul 02 21:41:04.740303 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.740251    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="be581cf6-68df-41db-8a1a-e8a269a742fe" containerName="storage-initializer"
Jul 02 21:41:04.740403 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.740307    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be581cf6-68df-41db-8a1a-e8a269a742fe" containerName="storage-initializer"
Jul 02 21:41:04.740403 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.740318    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="be581cf6-68df-41db-8a1a-e8a269a742fe" containerName="storage-initializer"
Jul 02 21:41:04.744879 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.744862    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:41:04.747558 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.747539    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 21:41:04.747669 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.747651    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 21:41:04.747737 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.747725    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-57xll\""
Jul 02 21:41:04.748716 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.748702    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-efdfd-predictor-serving-cert\""
Jul 02 21:41:04.748800 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.748715    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-efdfd-kube-rbac-proxy-sar-config\""
Jul 02 21:41:04.748800 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.748740    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 21:41:04.748800 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.748716    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 21:41:04.754513 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.754491    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"]
Jul 02 21:41:04.786161 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.786139    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:41:04.897175 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.897151    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bed40e68-dadf-47b8-b904-47226e33c87a-kserve-provision-location\") pod \"raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt\" (UID: \"bed40e68-dadf-47b8-b904-47226e33c87a\") " pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:41:04.897262 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.897205    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bed40e68-dadf-47b8-b904-47226e33c87a-cabundle-cert\") pod \"raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt\" (UID: \"bed40e68-dadf-47b8-b904-47226e33c87a\") " pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:41:04.897262 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.897227    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bed40e68-dadf-47b8-b904-47226e33c87a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt\" (UID: \"bed40e68-dadf-47b8-b904-47226e33c87a\") " pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:41:04.897262 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.897250    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bed40e68-dadf-47b8-b904-47226e33c87a-proxy-tls\") pod \"raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt\" (UID: \"bed40e68-dadf-47b8-b904-47226e33c87a\") " pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:41:04.897385 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.897267    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwj2x\" (UniqueName: \"kubernetes.io/projected/bed40e68-dadf-47b8-b904-47226e33c87a-kube-api-access-fwj2x\") pod \"raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt\" (UID: \"bed40e68-dadf-47b8-b904-47226e33c87a\") " pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:41:04.998475 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.998424    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwj2x\" (UniqueName: \"kubernetes.io/projected/bed40e68-dadf-47b8-b904-47226e33c87a-kube-api-access-fwj2x\") pod \"raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt\" (UID: \"bed40e68-dadf-47b8-b904-47226e33c87a\") " pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:41:04.998475 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.998472    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bed40e68-dadf-47b8-b904-47226e33c87a-kserve-provision-location\") pod \"raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt\" (UID: \"bed40e68-dadf-47b8-b904-47226e33c87a\") " pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:41:04.998607 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.998500    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bed40e68-dadf-47b8-b904-47226e33c87a-cabundle-cert\") pod \"raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt\" (UID: \"bed40e68-dadf-47b8-b904-47226e33c87a\") " pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:41:04.998607 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.998531    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bed40e68-dadf-47b8-b904-47226e33c87a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt\" (UID: \"bed40e68-dadf-47b8-b904-47226e33c87a\") " pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:41:04.998607 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.998564    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bed40e68-dadf-47b8-b904-47226e33c87a-proxy-tls\") pod \"raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt\" (UID: \"bed40e68-dadf-47b8-b904-47226e33c87a\") " pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:41:04.998895 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.998874    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bed40e68-dadf-47b8-b904-47226e33c87a-kserve-provision-location\") pod \"raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt\" (UID: \"bed40e68-dadf-47b8-b904-47226e33c87a\") " pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:41:04.999141 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.999119    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bed40e68-dadf-47b8-b904-47226e33c87a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt\" (UID: \"bed40e68-dadf-47b8-b904-47226e33c87a\") " pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:41:04.999242 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:04.999220    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bed40e68-dadf-47b8-b904-47226e33c87a-cabundle-cert\") pod \"raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt\" (UID: \"bed40e68-dadf-47b8-b904-47226e33c87a\") " pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:41:05.001043 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:05.001024    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bed40e68-dadf-47b8-b904-47226e33c87a-proxy-tls\") pod \"raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt\" (UID: \"bed40e68-dadf-47b8-b904-47226e33c87a\") " pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:41:05.007325 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:05.007304    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwj2x\" (UniqueName: \"kubernetes.io/projected/bed40e68-dadf-47b8-b904-47226e33c87a-kube-api-access-fwj2x\") pod \"raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt\" (UID: \"bed40e68-dadf-47b8-b904-47226e33c87a\") " pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:41:05.055701 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:05.055682    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:41:05.189408 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:05.189384    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"]
Jul 02 21:41:05.191692 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:41:05.191667    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed40e68_dadf_47b8_b904_47226e33c87a.slice/crio-51d8a62c62067a3c3e3d9e163fc9d5b06a53f1aeb0a04cc78d36dc0b0ed5d760 WatchSource:0}: Error finding container 51d8a62c62067a3c3e3d9e163fc9d5b06a53f1aeb0a04cc78d36dc0b0ed5d760: Status 404 returned error can't find the container with id 51d8a62c62067a3c3e3d9e163fc9d5b06a53f1aeb0a04cc78d36dc0b0ed5d760
Jul 02 21:41:05.554315 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:05.554279    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt" event={"ID":"bed40e68-dadf-47b8-b904-47226e33c87a","Type":"ContainerStarted","Data":"09a5ba4c097e805ba7fa7a07035ced8123e9bfc5b13cf6a73bacadf67ad72c62"}
Jul 02 21:41:05.554505 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:05.554334    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt" event={"ID":"bed40e68-dadf-47b8-b904-47226e33c87a","Type":"ContainerStarted","Data":"51d8a62c62067a3c3e3d9e163fc9d5b06a53f1aeb0a04cc78d36dc0b0ed5d760"}
Jul 02 21:41:09.573573 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:09.573537    2575 generic.go:356] "Generic (PLEG): container finished" podID="bed40e68-dadf-47b8-b904-47226e33c87a" containerID="09a5ba4c097e805ba7fa7a07035ced8123e9bfc5b13cf6a73bacadf67ad72c62" exitCode=0
Jul 02 21:41:09.573989 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:09.573609    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt" event={"ID":"bed40e68-dadf-47b8-b904-47226e33c87a","Type":"ContainerDied","Data":"09a5ba4c097e805ba7fa7a07035ced8123e9bfc5b13cf6a73bacadf67ad72c62"}
Jul 02 21:41:10.580007 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:10.579971    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt" event={"ID":"bed40e68-dadf-47b8-b904-47226e33c87a","Type":"ContainerStarted","Data":"3204fdd18ca28fc871d8e836f37072b9ad68f864fb83f5336b9aee4ae24f080f"}
Jul 02 21:41:10.580374 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:10.580039    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt" event={"ID":"bed40e68-dadf-47b8-b904-47226e33c87a","Type":"ContainerStarted","Data":"e8e3e687c70a9cd48a4b411bc0e072941a72403d423a126486fb20adba7869cf"}
Jul 02 21:41:10.580374 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:10.580223    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:41:10.601941 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:10.601895    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt" podStartSLOduration=2.8121004960000002 podStartE2EDuration="6.601880183s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.789779687s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:41:10.600006437 +0000 UTC m=+1342.226508313" watchObservedRunningTime="2026-07-02 21:41:10.601880183 +0000 UTC m=+1342.228382058"
Jul 02 21:41:11.584369 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:11.584340    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:41:11.585733 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:11.585706    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt" podUID="bed40e68-dadf-47b8-b904-47226e33c87a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 02 21:41:12.588372 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:12.588335    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt" podUID="bed40e68-dadf-47b8-b904-47226e33c87a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 02 21:41:17.593505 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:17.593474    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:41:17.594053 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:17.594028    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt" podUID="bed40e68-dadf-47b8-b904-47226e33c87a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 02 21:41:27.594752 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:27.594712    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt" podUID="bed40e68-dadf-47b8-b904-47226e33c87a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 02 21:41:37.594010 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:37.593965    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt" podUID="bed40e68-dadf-47b8-b904-47226e33c87a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 02 21:41:47.594317 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:47.594274    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt" podUID="bed40e68-dadf-47b8-b904-47226e33c87a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 02 21:41:57.594748 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:41:57.594709    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt" podUID="bed40e68-dadf-47b8-b904-47226e33c87a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 02 21:42:07.595005 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:07.594966    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt" podUID="bed40e68-dadf-47b8-b904-47226e33c87a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 02 21:42:17.594338 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:17.594295    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt" podUID="bed40e68-dadf-47b8-b904-47226e33c87a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 02 21:42:17.713119 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:17.713045    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:42:27.595144 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:27.595113    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:42:34.862268 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:34.862233    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"]
Jul 02 21:42:34.862666 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:34.862527    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt" podUID="bed40e68-dadf-47b8-b904-47226e33c87a" containerName="kserve-container" containerID="cri-o://e8e3e687c70a9cd48a4b411bc0e072941a72403d423a126486fb20adba7869cf" gracePeriod=30
Jul 02 21:42:34.862666 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:34.862589    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt" podUID="bed40e68-dadf-47b8-b904-47226e33c87a" containerName="kube-rbac-proxy" containerID="cri-o://3204fdd18ca28fc871d8e836f37072b9ad68f864fb83f5336b9aee4ae24f080f" gracePeriod=30
Jul 02 21:42:34.889085 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:34.889061    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:42:35.931458 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:35.931427    2575 generic.go:356] "Generic (PLEG): container finished" podID="bed40e68-dadf-47b8-b904-47226e33c87a" containerID="3204fdd18ca28fc871d8e836f37072b9ad68f864fb83f5336b9aee4ae24f080f" exitCode=2
Jul 02 21:42:35.931822 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:35.931500    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt" event={"ID":"bed40e68-dadf-47b8-b904-47226e33c87a","Type":"ContainerDied","Data":"3204fdd18ca28fc871d8e836f37072b9ad68f864fb83f5336b9aee4ae24f080f"}
Jul 02 21:42:37.589410 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:37.589370    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt" podUID="bed40e68-dadf-47b8-b904-47226e33c87a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Jul 02 21:42:37.594778 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:37.594717    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt" podUID="bed40e68-dadf-47b8-b904-47226e33c87a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 02 21:42:39.302855 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.302833    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:42:39.355228 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.355201    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bed40e68-dadf-47b8-b904-47226e33c87a-kube-rbac-proxy-sar-config\") pod \"bed40e68-dadf-47b8-b904-47226e33c87a\" (UID: \"bed40e68-dadf-47b8-b904-47226e33c87a\") "
Jul 02 21:42:39.355342 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.355232    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bed40e68-dadf-47b8-b904-47226e33c87a-proxy-tls\") pod \"bed40e68-dadf-47b8-b904-47226e33c87a\" (UID: \"bed40e68-dadf-47b8-b904-47226e33c87a\") "
Jul 02 21:42:39.355342 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.355265    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bed40e68-dadf-47b8-b904-47226e33c87a-cabundle-cert\") pod \"bed40e68-dadf-47b8-b904-47226e33c87a\" (UID: \"bed40e68-dadf-47b8-b904-47226e33c87a\") "
Jul 02 21:42:39.355431 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.355389    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwj2x\" (UniqueName: \"kubernetes.io/projected/bed40e68-dadf-47b8-b904-47226e33c87a-kube-api-access-fwj2x\") pod \"bed40e68-dadf-47b8-b904-47226e33c87a\" (UID: \"bed40e68-dadf-47b8-b904-47226e33c87a\") "
Jul 02 21:42:39.355488 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.355444    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bed40e68-dadf-47b8-b904-47226e33c87a-kserve-provision-location\") pod \"bed40e68-dadf-47b8-b904-47226e33c87a\" (UID: \"bed40e68-dadf-47b8-b904-47226e33c87a\") "
Jul 02 21:42:39.355619 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.355591    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed40e68-dadf-47b8-b904-47226e33c87a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bed40e68-dadf-47b8-b904-47226e33c87a" (UID: "bed40e68-dadf-47b8-b904-47226e33c87a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:42:39.355619 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.355595    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed40e68-dadf-47b8-b904-47226e33c87a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bed40e68-dadf-47b8-b904-47226e33c87a" (UID: "bed40e68-dadf-47b8-b904-47226e33c87a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:42:39.355794 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.355767    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed40e68-dadf-47b8-b904-47226e33c87a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bed40e68-dadf-47b8-b904-47226e33c87a" (UID: "bed40e68-dadf-47b8-b904-47226e33c87a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:42:39.355857 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.355821    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bed40e68-dadf-47b8-b904-47226e33c87a-cabundle-cert\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:42:39.355910 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.355865    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bed40e68-dadf-47b8-b904-47226e33c87a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:42:39.357789 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.357762    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed40e68-dadf-47b8-b904-47226e33c87a-kube-api-access-fwj2x" (OuterVolumeSpecName: "kube-api-access-fwj2x") pod "bed40e68-dadf-47b8-b904-47226e33c87a" (UID: "bed40e68-dadf-47b8-b904-47226e33c87a"). InnerVolumeSpecName "kube-api-access-fwj2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:42:39.357789 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.357762    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed40e68-dadf-47b8-b904-47226e33c87a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bed40e68-dadf-47b8-b904-47226e33c87a" (UID: "bed40e68-dadf-47b8-b904-47226e33c87a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:42:39.456243 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.456159    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwj2x\" (UniqueName: \"kubernetes.io/projected/bed40e68-dadf-47b8-b904-47226e33c87a-kube-api-access-fwj2x\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:42:39.456243 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.456209    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bed40e68-dadf-47b8-b904-47226e33c87a-kserve-provision-location\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:42:39.456243 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.456220    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bed40e68-dadf-47b8-b904-47226e33c87a-proxy-tls\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:42:39.949577 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.949547    2575 generic.go:356] "Generic (PLEG): container finished" podID="bed40e68-dadf-47b8-b904-47226e33c87a" containerID="e8e3e687c70a9cd48a4b411bc0e072941a72403d423a126486fb20adba7869cf" exitCode=0
Jul 02 21:42:39.949772 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.949615    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"
Jul 02 21:42:39.949772 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.949621    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt" event={"ID":"bed40e68-dadf-47b8-b904-47226e33c87a","Type":"ContainerDied","Data":"e8e3e687c70a9cd48a4b411bc0e072941a72403d423a126486fb20adba7869cf"}
Jul 02 21:42:39.949772 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.949683    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt" event={"ID":"bed40e68-dadf-47b8-b904-47226e33c87a","Type":"ContainerDied","Data":"51d8a62c62067a3c3e3d9e163fc9d5b06a53f1aeb0a04cc78d36dc0b0ed5d760"}
Jul 02 21:42:39.949772 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.949711    2575 scope.go:117] "RemoveContainer" containerID="3204fdd18ca28fc871d8e836f37072b9ad68f864fb83f5336b9aee4ae24f080f"
Jul 02 21:42:39.953346 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.953306    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:42:39.961988 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.961967    2575 scope.go:117] "RemoveContainer" containerID="e8e3e687c70a9cd48a4b411bc0e072941a72403d423a126486fb20adba7869cf"
Jul 02 21:42:39.969689 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.969667    2575 scope.go:117] "RemoveContainer" containerID="09a5ba4c097e805ba7fa7a07035ced8123e9bfc5b13cf6a73bacadf67ad72c62"
Jul 02 21:42:39.973765 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.973744    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"]
Jul 02 21:42:39.975954 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.975927    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-efdfd-predictor-7fdc445bdf-4z5bt"]
Jul 02 21:42:39.978016 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.977998    2575 scope.go:117] "RemoveContainer" containerID="3204fdd18ca28fc871d8e836f37072b9ad68f864fb83f5336b9aee4ae24f080f"
Jul 02 21:42:39.978296 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:42:39.978273    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3204fdd18ca28fc871d8e836f37072b9ad68f864fb83f5336b9aee4ae24f080f\": container with ID starting with 3204fdd18ca28fc871d8e836f37072b9ad68f864fb83f5336b9aee4ae24f080f not found: ID does not exist" containerID="3204fdd18ca28fc871d8e836f37072b9ad68f864fb83f5336b9aee4ae24f080f"
Jul 02 21:42:39.978351 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.978308    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3204fdd18ca28fc871d8e836f37072b9ad68f864fb83f5336b9aee4ae24f080f"} err="failed to get container status \"3204fdd18ca28fc871d8e836f37072b9ad68f864fb83f5336b9aee4ae24f080f\": rpc error: code = NotFound desc = could not find container \"3204fdd18ca28fc871d8e836f37072b9ad68f864fb83f5336b9aee4ae24f080f\": container with ID starting with 3204fdd18ca28fc871d8e836f37072b9ad68f864fb83f5336b9aee4ae24f080f not found: ID does not exist"
Jul 02 21:42:39.978351 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.978334    2575 scope.go:117] "RemoveContainer" containerID="e8e3e687c70a9cd48a4b411bc0e072941a72403d423a126486fb20adba7869cf"
Jul 02 21:42:39.978626 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:42:39.978607    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e3e687c70a9cd48a4b411bc0e072941a72403d423a126486fb20adba7869cf\": container with ID starting with e8e3e687c70a9cd48a4b411bc0e072941a72403d423a126486fb20adba7869cf not found: ID does not exist" containerID="e8e3e687c70a9cd48a4b411bc0e072941a72403d423a126486fb20adba7869cf"
Jul 02 21:42:39.978675 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.978630    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e3e687c70a9cd48a4b411bc0e072941a72403d423a126486fb20adba7869cf"} err="failed to get container status \"e8e3e687c70a9cd48a4b411bc0e072941a72403d423a126486fb20adba7869cf\": rpc error: code = NotFound desc = could not find container \"e8e3e687c70a9cd48a4b411bc0e072941a72403d423a126486fb20adba7869cf\": container with ID starting with e8e3e687c70a9cd48a4b411bc0e072941a72403d423a126486fb20adba7869cf not found: ID does not exist"
Jul 02 21:42:39.978675 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.978648    2575 scope.go:117] "RemoveContainer" containerID="09a5ba4c097e805ba7fa7a07035ced8123e9bfc5b13cf6a73bacadf67ad72c62"
Jul 02 21:42:39.978890 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:42:39.978875    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09a5ba4c097e805ba7fa7a07035ced8123e9bfc5b13cf6a73bacadf67ad72c62\": container with ID starting with 09a5ba4c097e805ba7fa7a07035ced8123e9bfc5b13cf6a73bacadf67ad72c62 not found: ID does not exist" containerID="09a5ba4c097e805ba7fa7a07035ced8123e9bfc5b13cf6a73bacadf67ad72c62"
Jul 02 21:42:39.978932 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:39.978893    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09a5ba4c097e805ba7fa7a07035ced8123e9bfc5b13cf6a73bacadf67ad72c62"} err="failed to get container status \"09a5ba4c097e805ba7fa7a07035ced8123e9bfc5b13cf6a73bacadf67ad72c62\": rpc error: code = NotFound desc = could not find container \"09a5ba4c097e805ba7fa7a07035ced8123e9bfc5b13cf6a73bacadf67ad72c62\": container with ID starting with 09a5ba4c097e805ba7fa7a07035ced8123e9bfc5b13cf6a73bacadf67ad72c62 not found: ID does not exist"
Jul 02 21:42:40.011005 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.010974    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"]
Jul 02 21:42:40.011333 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.011319    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bed40e68-dadf-47b8-b904-47226e33c87a" containerName="kube-rbac-proxy"
Jul 02 21:42:40.011380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.011336    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bed40e68-dadf-47b8-b904-47226e33c87a" containerName="kube-rbac-proxy"
Jul 02 21:42:40.011380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.011356    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bed40e68-dadf-47b8-b904-47226e33c87a" containerName="storage-initializer"
Jul 02 21:42:40.011380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.011363    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bed40e68-dadf-47b8-b904-47226e33c87a" containerName="storage-initializer"
Jul 02 21:42:40.011380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.011375    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bed40e68-dadf-47b8-b904-47226e33c87a" containerName="kserve-container"
Jul 02 21:42:40.011380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.011380    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bed40e68-dadf-47b8-b904-47226e33c87a" containerName="kserve-container"
Jul 02 21:42:40.011531 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.011439    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bed40e68-dadf-47b8-b904-47226e33c87a" containerName="kserve-container"
Jul 02 21:42:40.011531 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.011450    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bed40e68-dadf-47b8-b904-47226e33c87a" containerName="kube-rbac-proxy"
Jul 02 21:42:40.015752 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.015735    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:42:40.019250 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.019227    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 21:42:40.019578 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.019557    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 21:42:40.019698 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.019567    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 21:42:40.019698 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.019607    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-9d762-predictor-serving-cert\""
Jul 02 21:42:40.019698 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.019629    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-57xll\""
Jul 02 21:42:40.019956 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.019942    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-9d762-kube-rbac-proxy-sar-config\""
Jul 02 21:42:40.020014 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.020000    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 21:42:40.032299 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.032265    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"]
Jul 02 21:42:40.161919 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.161876    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-cabundle-cert\") pod \"raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b\" (UID: \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:42:40.162104 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.161940    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdx89\" (UniqueName: \"kubernetes.io/projected/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-kube-api-access-cdx89\") pod \"raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b\" (UID: \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:42:40.162104 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.161971    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-proxy-tls\") pod \"raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b\" (UID: \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:42:40.162104 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.162052    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b\" (UID: \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:42:40.162263 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.162104    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-kserve-provision-location\") pod \"raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b\" (UID: \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:42:40.262685 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.262571    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-cabundle-cert\") pod \"raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b\" (UID: \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:42:40.262685 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.262666    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdx89\" (UniqueName: \"kubernetes.io/projected/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-kube-api-access-cdx89\") pod \"raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b\" (UID: \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:42:40.262931 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.262696    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-proxy-tls\") pod \"raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b\" (UID: \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:42:40.262931 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.262744    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b\" (UID: \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:42:40.262931 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.262778    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-kserve-provision-location\") pod \"raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b\" (UID: \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:42:40.262931 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:42:40.262841    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-serving-cert: secret "raw-sklearn-runtime-9d762-predictor-serving-cert" not found
Jul 02 21:42:40.262931 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:42:40.262929    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-proxy-tls podName:33beb7cc-d071-4ccb-8c5b-e63577ffb3c4 nodeName:}" failed. No retries permitted until 2026-07-02 21:42:40.762908998 +0000 UTC m=+1432.389410856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-proxy-tls") pod "raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b" (UID: "33beb7cc-d071-4ccb-8c5b-e63577ffb3c4") : secret "raw-sklearn-runtime-9d762-predictor-serving-cert" not found
Jul 02 21:42:40.263255 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.263233    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-kserve-provision-location\") pod \"raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b\" (UID: \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:42:40.263431 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.263408    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-cabundle-cert\") pod \"raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b\" (UID: \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:42:40.263493 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.263457    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b\" (UID: \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:42:40.272030 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.272004    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdx89\" (UniqueName: \"kubernetes.io/projected/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-kube-api-access-cdx89\") pod \"raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b\" (UID: \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:42:40.500160 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.500125    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:42:40.767485 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.767450    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-proxy-tls\") pod \"raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b\" (UID: \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:42:40.769741 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.769713    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-proxy-tls\") pod \"raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b\" (UID: \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:42:40.934575 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:40.934531    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:42:41.015015 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:41.014756    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bed40e68-dadf-47b8-b904-47226e33c87a" path="/var/lib/kubelet/pods/bed40e68-dadf-47b8-b904-47226e33c87a/volumes"
Jul 02 21:42:41.067328 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:41.067258    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"]
Jul 02 21:42:41.069391 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:42:41.069364    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33beb7cc_d071_4ccb_8c5b_e63577ffb3c4.slice/crio-c2a03efade36b45238fa1b98ecd5436000301f97b99206db69206607fac1ab98 WatchSource:0}: Error finding container c2a03efade36b45238fa1b98ecd5436000301f97b99206db69206607fac1ab98: Status 404 returned error can't find the container with id c2a03efade36b45238fa1b98ecd5436000301f97b99206db69206607fac1ab98
Jul 02 21:42:41.960849 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:41.960806    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b" event={"ID":"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4","Type":"ContainerStarted","Data":"d85a3151f10bbf7c57a7993e7dadc13a5e8db3f2ca5efb58a8f0cf84ad536e07"}
Jul 02 21:42:41.961250 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:41.960897    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b" event={"ID":"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4","Type":"ContainerStarted","Data":"c2a03efade36b45238fa1b98ecd5436000301f97b99206db69206607fac1ab98"}
Jul 02 21:42:44.976670 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:44.976638    2575 generic.go:356] "Generic (PLEG): container finished" podID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerID="d85a3151f10bbf7c57a7993e7dadc13a5e8db3f2ca5efb58a8f0cf84ad536e07" exitCode=0
Jul 02 21:42:44.977058 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:44.976696    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b" event={"ID":"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4","Type":"ContainerDied","Data":"d85a3151f10bbf7c57a7993e7dadc13a5e8db3f2ca5efb58a8f0cf84ad536e07"}
Jul 02 21:42:45.982900 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:45.982869    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b" event={"ID":"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4","Type":"ContainerStarted","Data":"a2fe34f2ab8b6715f3f610d36c022a9f6d0db33218bf1f554ef27e483fd06c4e"}
Jul 02 21:42:45.983392 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:45.982939    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b" event={"ID":"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4","Type":"ContainerStarted","Data":"f65556a2d079cafbb7e2ae8e280fa5d7c1be59ace8dac1d466af8b645fb24d93"}
Jul 02 21:42:45.983392 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:45.983159    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:42:46.004609 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:46.004554    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b" podStartSLOduration=3.512452976 podStartE2EDuration="7.004535489s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.492082513s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:42:46.002196874 +0000 UTC m=+1437.628698770" watchObservedRunningTime="2026-07-02 21:42:46.004535489 +0000 UTC m=+1437.631037369"
Jul 02 21:42:46.986749 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:46.986716    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:42:46.988228 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:46.988194    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b" podUID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 21:42:47.990538 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:47.990489    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b" podUID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 21:42:52.995464 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:52.995436    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:42:52.995893 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:42:52.995868    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b" podUID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 21:43:02.996813 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:43:02.996774    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b" podUID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 21:43:12.996698 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:43:12.996659    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b" podUID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 21:43:22.996822 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:43:22.996784    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b" podUID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 21:43:32.996266 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:43:32.996225    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b" podUID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 21:43:42.996865 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:43:42.996827    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b" podUID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 21:43:52.996401 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:43:52.996362    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b" podUID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 02 21:43:59.015054 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:43:59.015024    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:44:00.087205 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:00.087158    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"]
Jul 02 21:44:00.267997 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:00.267946    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b" podUID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerName="kserve-container" containerID="cri-o://f65556a2d079cafbb7e2ae8e280fa5d7c1be59ace8dac1d466af8b645fb24d93" gracePeriod=30
Jul 02 21:44:00.268229 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:00.267963    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b" podUID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerName="kube-rbac-proxy" containerID="cri-o://a2fe34f2ab8b6715f3f610d36c022a9f6d0db33218bf1f554ef27e483fd06c4e" gracePeriod=30
Jul 02 21:44:00.840195 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:00.840144    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:44:01.275103 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:01.275071    2575 generic.go:356] "Generic (PLEG): container finished" podID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerID="a2fe34f2ab8b6715f3f610d36c022a9f6d0db33218bf1f554ef27e483fd06c4e" exitCode=2
Jul 02 21:44:01.275487 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:01.275149    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b" event={"ID":"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4","Type":"ContainerDied","Data":"a2fe34f2ab8b6715f3f610d36c022a9f6d0db33218bf1f554ef27e483fd06c4e"}
Jul 02 21:44:02.991258 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:02.991217    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b" podUID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Jul 02 21:44:04.504113 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:04.504089    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:44:04.636891 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:04.636797    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-kserve-provision-location\") pod \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\" (UID: \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\") "
Jul 02 21:44:04.636891 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:04.636857    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-proxy-tls\") pod \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\" (UID: \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\") "
Jul 02 21:44:04.637226 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:04.636939    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdx89\" (UniqueName: \"kubernetes.io/projected/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-kube-api-access-cdx89\") pod \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\" (UID: \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\") "
Jul 02 21:44:04.637226 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:04.636962    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-kube-rbac-proxy-sar-config\") pod \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\" (UID: \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\") "
Jul 02 21:44:04.637226 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:04.637046    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-cabundle-cert\") pod \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\" (UID: \"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4\") "
Jul 02 21:44:04.637226 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:04.637191    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" (UID: "33beb7cc-d071-4ccb-8c5b-e63577ffb3c4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:44:04.637422 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:04.637310    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-kserve-provision-location\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:44:04.637461 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:04.637433    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" (UID: "33beb7cc-d071-4ccb-8c5b-e63577ffb3c4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:44:04.637496 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:04.637439    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" (UID: "33beb7cc-d071-4ccb-8c5b-e63577ffb3c4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:44:04.638963 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:04.638941    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-kube-api-access-cdx89" (OuterVolumeSpecName: "kube-api-access-cdx89") pod "33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" (UID: "33beb7cc-d071-4ccb-8c5b-e63577ffb3c4"). InnerVolumeSpecName "kube-api-access-cdx89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:44:04.639046 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:04.639002    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" (UID: "33beb7cc-d071-4ccb-8c5b-e63577ffb3c4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:44:04.737929 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:04.737900    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdx89\" (UniqueName: \"kubernetes.io/projected/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-kube-api-access-cdx89\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:44:04.737929 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:04.737925    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:44:04.737929 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:04.737935    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-cabundle-cert\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:44:04.738099 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:04.737943    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4-proxy-tls\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:44:05.293008 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:05.292936    2575 generic.go:356] "Generic (PLEG): container finished" podID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerID="f65556a2d079cafbb7e2ae8e280fa5d7c1be59ace8dac1d466af8b645fb24d93" exitCode=0
Jul 02 21:44:05.293008 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:05.292987    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"
Jul 02 21:44:05.293008 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:05.292995    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b" event={"ID":"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4","Type":"ContainerDied","Data":"f65556a2d079cafbb7e2ae8e280fa5d7c1be59ace8dac1d466af8b645fb24d93"}
Jul 02 21:44:05.293290 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:05.293051    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b" event={"ID":"33beb7cc-d071-4ccb-8c5b-e63577ffb3c4","Type":"ContainerDied","Data":"c2a03efade36b45238fa1b98ecd5436000301f97b99206db69206607fac1ab98"}
Jul 02 21:44:05.293290 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:05.293089    2575 scope.go:117] "RemoveContainer" containerID="a2fe34f2ab8b6715f3f610d36c022a9f6d0db33218bf1f554ef27e483fd06c4e"
Jul 02 21:44:05.300943 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:05.300923    2575 scope.go:117] "RemoveContainer" containerID="f65556a2d079cafbb7e2ae8e280fa5d7c1be59ace8dac1d466af8b645fb24d93"
Jul 02 21:44:05.308035 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:05.308017    2575 scope.go:117] "RemoveContainer" containerID="d85a3151f10bbf7c57a7993e7dadc13a5e8db3f2ca5efb58a8f0cf84ad536e07"
Jul 02 21:44:05.312901 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:05.312880    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"]
Jul 02 21:44:05.315433 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:05.315419    2575 scope.go:117] "RemoveContainer" containerID="a2fe34f2ab8b6715f3f610d36c022a9f6d0db33218bf1f554ef27e483fd06c4e"
Jul 02 21:44:05.315688 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:44:05.315660    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2fe34f2ab8b6715f3f610d36c022a9f6d0db33218bf1f554ef27e483fd06c4e\": container with ID starting with a2fe34f2ab8b6715f3f610d36c022a9f6d0db33218bf1f554ef27e483fd06c4e not found: ID does not exist" containerID="a2fe34f2ab8b6715f3f610d36c022a9f6d0db33218bf1f554ef27e483fd06c4e"
Jul 02 21:44:05.315769 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:05.315689    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2fe34f2ab8b6715f3f610d36c022a9f6d0db33218bf1f554ef27e483fd06c4e"} err="failed to get container status \"a2fe34f2ab8b6715f3f610d36c022a9f6d0db33218bf1f554ef27e483fd06c4e\": rpc error: code = NotFound desc = could not find container \"a2fe34f2ab8b6715f3f610d36c022a9f6d0db33218bf1f554ef27e483fd06c4e\": container with ID starting with a2fe34f2ab8b6715f3f610d36c022a9f6d0db33218bf1f554ef27e483fd06c4e not found: ID does not exist"
Jul 02 21:44:05.315769 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:05.315712    2575 scope.go:117] "RemoveContainer" containerID="f65556a2d079cafbb7e2ae8e280fa5d7c1be59ace8dac1d466af8b645fb24d93"
Jul 02 21:44:05.315986 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:44:05.315962    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f65556a2d079cafbb7e2ae8e280fa5d7c1be59ace8dac1d466af8b645fb24d93\": container with ID starting with f65556a2d079cafbb7e2ae8e280fa5d7c1be59ace8dac1d466af8b645fb24d93 not found: ID does not exist" containerID="f65556a2d079cafbb7e2ae8e280fa5d7c1be59ace8dac1d466af8b645fb24d93"
Jul 02 21:44:05.316040 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:05.315996    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f65556a2d079cafbb7e2ae8e280fa5d7c1be59ace8dac1d466af8b645fb24d93"} err="failed to get container status \"f65556a2d079cafbb7e2ae8e280fa5d7c1be59ace8dac1d466af8b645fb24d93\": rpc error: code = NotFound desc = could not find container \"f65556a2d079cafbb7e2ae8e280fa5d7c1be59ace8dac1d466af8b645fb24d93\": container with ID starting with f65556a2d079cafbb7e2ae8e280fa5d7c1be59ace8dac1d466af8b645fb24d93 not found: ID does not exist"
Jul 02 21:44:05.316040 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:05.316013    2575 scope.go:117] "RemoveContainer" containerID="d85a3151f10bbf7c57a7993e7dadc13a5e8db3f2ca5efb58a8f0cf84ad536e07"
Jul 02 21:44:05.316307 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:44:05.316282    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d85a3151f10bbf7c57a7993e7dadc13a5e8db3f2ca5efb58a8f0cf84ad536e07\": container with ID starting with d85a3151f10bbf7c57a7993e7dadc13a5e8db3f2ca5efb58a8f0cf84ad536e07 not found: ID does not exist" containerID="d85a3151f10bbf7c57a7993e7dadc13a5e8db3f2ca5efb58a8f0cf84ad536e07"
Jul 02 21:44:05.316380 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:05.316314    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d85a3151f10bbf7c57a7993e7dadc13a5e8db3f2ca5efb58a8f0cf84ad536e07"} err="failed to get container status \"d85a3151f10bbf7c57a7993e7dadc13a5e8db3f2ca5efb58a8f0cf84ad536e07\": rpc error: code = NotFound desc = could not find container \"d85a3151f10bbf7c57a7993e7dadc13a5e8db3f2ca5efb58a8f0cf84ad536e07\": container with ID starting with d85a3151f10bbf7c57a7993e7dadc13a5e8db3f2ca5efb58a8f0cf84ad536e07 not found: ID does not exist"
Jul 02 21:44:05.317989 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:05.317967    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9d762-predictor-76c5cffb77-6968b"]
Jul 02 21:44:07.015701 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.015670    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" path="/var/lib/kubelet/pods/33beb7cc-d071-4ccb-8c5b-e63577ffb3c4/volumes"
Jul 02 21:44:07.274292 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.274222    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gx5nx/must-gather-zhcgp"]
Jul 02 21:44:07.274541 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.274528    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerName="kube-rbac-proxy"
Jul 02 21:44:07.274541 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.274542    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerName="kube-rbac-proxy"
Jul 02 21:44:07.274623 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.274555    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerName="storage-initializer"
Jul 02 21:44:07.274623 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.274561    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerName="storage-initializer"
Jul 02 21:44:07.274623 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.274590    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerName="kserve-container"
Jul 02 21:44:07.274623 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.274596    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerName="kserve-container"
Jul 02 21:44:07.274751 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.274644    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerName="kserve-container"
Jul 02 21:44:07.274751 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.274656    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="33beb7cc-d071-4ccb-8c5b-e63577ffb3c4" containerName="kube-rbac-proxy"
Jul 02 21:44:07.279037 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.279016    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gx5nx/must-gather-zhcgp"
Jul 02 21:44:07.279717 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.279694    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:44:07.281657 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.281634    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gx5nx\"/\"openshift-service-ca.crt\""
Jul 02 21:44:07.281768 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.281717    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gx5nx\"/\"kube-root-ca.crt\""
Jul 02 21:44:07.282828 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.282813    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gx5nx\"/\"default-dockercfg-szb6b\""
Jul 02 21:44:07.286970 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.286950    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gx5nx/must-gather-zhcgp"]
Jul 02 21:44:07.345070 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.345051    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:44:07.458914 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.458891    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckct8\" (UniqueName: \"kubernetes.io/projected/d7a33511-5d26-4719-a154-f436ab35b0cd-kube-api-access-ckct8\") pod \"must-gather-zhcgp\" (UID: \"d7a33511-5d26-4719-a154-f436ab35b0cd\") " pod="openshift-must-gather-gx5nx/must-gather-zhcgp"
Jul 02 21:44:07.459006 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.458946    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d7a33511-5d26-4719-a154-f436ab35b0cd-must-gather-output\") pod \"must-gather-zhcgp\" (UID: \"d7a33511-5d26-4719-a154-f436ab35b0cd\") " pod="openshift-must-gather-gx5nx/must-gather-zhcgp"
Jul 02 21:44:07.559680 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.559621    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckct8\" (UniqueName: \"kubernetes.io/projected/d7a33511-5d26-4719-a154-f436ab35b0cd-kube-api-access-ckct8\") pod \"must-gather-zhcgp\" (UID: \"d7a33511-5d26-4719-a154-f436ab35b0cd\") " pod="openshift-must-gather-gx5nx/must-gather-zhcgp"
Jul 02 21:44:07.559772 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.559686    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d7a33511-5d26-4719-a154-f436ab35b0cd-must-gather-output\") pod \"must-gather-zhcgp\" (UID: \"d7a33511-5d26-4719-a154-f436ab35b0cd\") " pod="openshift-must-gather-gx5nx/must-gather-zhcgp"
Jul 02 21:44:07.559944 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.559926    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d7a33511-5d26-4719-a154-f436ab35b0cd-must-gather-output\") pod \"must-gather-zhcgp\" (UID: \"d7a33511-5d26-4719-a154-f436ab35b0cd\") " pod="openshift-must-gather-gx5nx/must-gather-zhcgp"
Jul 02 21:44:07.568033 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.568011    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckct8\" (UniqueName: \"kubernetes.io/projected/d7a33511-5d26-4719-a154-f436ab35b0cd-kube-api-access-ckct8\") pod \"must-gather-zhcgp\" (UID: \"d7a33511-5d26-4719-a154-f436ab35b0cd\") " pod="openshift-must-gather-gx5nx/must-gather-zhcgp"
Jul 02 21:44:07.598925 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.598904    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gx5nx/must-gather-zhcgp"
Jul 02 21:44:07.720617 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.720593    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gx5nx/must-gather-zhcgp"]
Jul 02 21:44:07.722390 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:44:07.722355    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7a33511_5d26_4719_a154_f436ab35b0cd.slice/crio-0de324e486664a1bfe51a793672d1e818d50f985957e12ee69ca70115506de26 WatchSource:0}: Error finding container 0de324e486664a1bfe51a793672d1e818d50f985957e12ee69ca70115506de26: Status 404 returned error can't find the container with id 0de324e486664a1bfe51a793672d1e818d50f985957e12ee69ca70115506de26
Jul 02 21:44:07.724118 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:07.724101    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 21:44:08.311317 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:08.311277    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gx5nx/must-gather-zhcgp" event={"ID":"d7a33511-5d26-4719-a154-f436ab35b0cd","Type":"ContainerStarted","Data":"0de324e486664a1bfe51a793672d1e818d50f985957e12ee69ca70115506de26"}
Jul 02 21:44:14.340327 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:14.340285    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gx5nx/must-gather-zhcgp" event={"ID":"d7a33511-5d26-4719-a154-f436ab35b0cd","Type":"ContainerStarted","Data":"d610440baebf4c716d7f0101f53b5634d03f421132a7e39ac6602e1e64fd6424"}
Jul 02 21:44:14.340753 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:14.340346    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gx5nx/must-gather-zhcgp" event={"ID":"d7a33511-5d26-4719-a154-f436ab35b0cd","Type":"ContainerStarted","Data":"8bf97432dfb3cab66b9fabf7329a6cdc5f42b7188c2315e67d01d3ab56217914"}
Jul 02 21:44:14.357978 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:14.357919    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gx5nx/must-gather-zhcgp" podStartSLOduration=1.22756364 podStartE2EDuration="7.357897576s" totalImagesPullingTime="6.130333936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:44:14.356816889 +0000 UTC m=+1525.983318769" watchObservedRunningTime="2026-07-02 21:44:14.357897576 +0000 UTC m=+1525.984399452"
Jul 02 21:44:35.434719 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:35.434685    2575 generic.go:356] "Generic (PLEG): container finished" podID="d7a33511-5d26-4719-a154-f436ab35b0cd" containerID="8bf97432dfb3cab66b9fabf7329a6cdc5f42b7188c2315e67d01d3ab56217914" exitCode=0
Jul 02 21:44:35.435121 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:35.434760    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gx5nx/must-gather-zhcgp" event={"ID":"d7a33511-5d26-4719-a154-f436ab35b0cd","Type":"ContainerDied","Data":"8bf97432dfb3cab66b9fabf7329a6cdc5f42b7188c2315e67d01d3ab56217914"}
Jul 02 21:44:35.435121 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:35.435068    2575 scope.go:117] "RemoveContainer" containerID="8bf97432dfb3cab66b9fabf7329a6cdc5f42b7188c2315e67d01d3ab56217914"
Jul 02 21:44:35.643711 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:35.643680    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-gx5nx_must-gather-zhcgp_d7a33511-5d26-4719-a154-f436ab35b0cd/gather/0.log"
Jul 02 21:44:36.394885 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:36.394858    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:44:36.603300 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:36.603273    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-thbg4/must-gather-9cgfm"]
Jul 02 21:44:36.605544 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:36.605518    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-thbg4/must-gather-9cgfm"
Jul 02 21:44:36.608265 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:36.608228    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:44:36.608385 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:36.608290    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-thbg4\"/\"default-dockercfg-ls6ns\""
Jul 02 21:44:36.608385 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:36.608292    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-thbg4\"/\"openshift-service-ca.crt\""
Jul 02 21:44:36.609811 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:36.609793    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-thbg4\"/\"kube-root-ca.crt\""
Jul 02 21:44:36.613344 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:36.613325    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-thbg4/must-gather-9cgfm"]
Jul 02 21:44:36.678254 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:36.678201    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:44:36.717615 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:36.717593    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3cc6a0dc-899c-4616-8c6d-9c6f445abad6-must-gather-output\") pod \"must-gather-9cgfm\" (UID: \"3cc6a0dc-899c-4616-8c6d-9c6f445abad6\") " pod="openshift-must-gather-thbg4/must-gather-9cgfm"
Jul 02 21:44:36.717694 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:36.717621    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttw6f\" (UniqueName: \"kubernetes.io/projected/3cc6a0dc-899c-4616-8c6d-9c6f445abad6-kube-api-access-ttw6f\") pod \"must-gather-9cgfm\" (UID: \"3cc6a0dc-899c-4616-8c6d-9c6f445abad6\") " pod="openshift-must-gather-thbg4/must-gather-9cgfm"
Jul 02 21:44:36.818318 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:36.818295    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3cc6a0dc-899c-4616-8c6d-9c6f445abad6-must-gather-output\") pod \"must-gather-9cgfm\" (UID: \"3cc6a0dc-899c-4616-8c6d-9c6f445abad6\") " pod="openshift-must-gather-thbg4/must-gather-9cgfm"
Jul 02 21:44:36.818403 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:36.818322    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttw6f\" (UniqueName: \"kubernetes.io/projected/3cc6a0dc-899c-4616-8c6d-9c6f445abad6-kube-api-access-ttw6f\") pod \"must-gather-9cgfm\" (UID: \"3cc6a0dc-899c-4616-8c6d-9c6f445abad6\") " pod="openshift-must-gather-thbg4/must-gather-9cgfm"
Jul 02 21:44:36.818632 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:36.818612    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3cc6a0dc-899c-4616-8c6d-9c6f445abad6-must-gather-output\") pod \"must-gather-9cgfm\" (UID: \"3cc6a0dc-899c-4616-8c6d-9c6f445abad6\") " pod="openshift-must-gather-thbg4/must-gather-9cgfm"
Jul 02 21:44:36.827060 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:36.827038    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttw6f\" (UniqueName: \"kubernetes.io/projected/3cc6a0dc-899c-4616-8c6d-9c6f445abad6-kube-api-access-ttw6f\") pod \"must-gather-9cgfm\" (UID: \"3cc6a0dc-899c-4616-8c6d-9c6f445abad6\") " pod="openshift-must-gather-thbg4/must-gather-9cgfm"
Jul 02 21:44:36.916083 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:36.916062    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-thbg4/must-gather-9cgfm"
Jul 02 21:44:37.037472 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:37.037431    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-thbg4/must-gather-9cgfm"]
Jul 02 21:44:37.039974 ip-10-0-135-227 kubenswrapper[2575]: W0702 21:44:37.039941    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cc6a0dc_899c_4616_8c6d_9c6f445abad6.slice/crio-2b1d4bc2c2026d84af49e4dd1854f17a26b4d727061d0b10029792aa75fa17e7 WatchSource:0}: Error finding container 2b1d4bc2c2026d84af49e4dd1854f17a26b4d727061d0b10029792aa75fa17e7: Status 404 returned error can't find the container with id 2b1d4bc2c2026d84af49e4dd1854f17a26b4d727061d0b10029792aa75fa17e7
Jul 02 21:44:37.443161 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:37.443129    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-thbg4/must-gather-9cgfm" event={"ID":"3cc6a0dc-899c-4616-8c6d-9c6f445abad6","Type":"ContainerStarted","Data":"2b1d4bc2c2026d84af49e4dd1854f17a26b4d727061d0b10029792aa75fa17e7"}
Jul 02 21:44:38.451008 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:38.450976    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-thbg4/must-gather-9cgfm" event={"ID":"3cc6a0dc-899c-4616-8c6d-9c6f445abad6","Type":"ContainerStarted","Data":"8f3b725089b8efdc2bea3d2d9954b95127402c0d259e50c96457b9aad12316a2"}
Jul 02 21:44:38.451455 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:38.451023    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-thbg4/must-gather-9cgfm" event={"ID":"3cc6a0dc-899c-4616-8c6d-9c6f445abad6","Type":"ContainerStarted","Data":"16c464bd7e9f48256ed6cfb813d139f6e7b347efdf332df6927198d6ffd8b3c9"}
Jul 02 21:44:38.468093 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:38.468036    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-thbg4/must-gather-9cgfm" podStartSLOduration=1.569986069 podStartE2EDuration="2.468016893s" totalImagesPullingTime="898.030824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:44:38.466979844 +0000 UTC m=+1550.093481724" watchObservedRunningTime="2026-07-02 21:44:38.468016893 +0000 UTC m=+1550.094518777"
Jul 02 21:44:39.286707 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:39.286681    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gpccf_3a6e6de1-8b1b-478e-8dce-996e81502c1e/global-pull-secret-syncer/0.log"
Jul 02 21:44:39.449324 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:39.449297    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log"
Jul 02 21:44:39.569677 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:39.569603    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-227.ec2.internal_c947105a64a6be554217ac8fde87371e/haproxy/0.log"
Jul 02 21:44:41.378905 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:41.378769    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-gx5nx/must-gather-zhcgp"]
Jul 02 21:44:41.379731 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:41.379584    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-gx5nx/must-gather-zhcgp" podUID="d7a33511-5d26-4719-a154-f436ab35b0cd" containerName="copy" containerID="cri-o://d610440baebf4c716d7f0101f53b5634d03f421132a7e39ac6602e1e64fd6424" gracePeriod=2
Jul 02 21:44:41.385390 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:41.385361    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-gx5nx/must-gather-zhcgp"]
Jul 02 21:44:41.764659 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:41.764629    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-gx5nx_must-gather-zhcgp_d7a33511-5d26-4719-a154-f436ab35b0cd/copy/0.log"
Jul 02 21:44:41.767632 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:41.767401    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gx5nx/must-gather-zhcgp"
Jul 02 21:44:41.769761 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:44:41.769718    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-zhcgp\" is forbidden: User \"system:node:ip-10-0-135-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-gx5nx\": no relationship found between node 'ip-10-0-135-227.ec2.internal' and this object" podUID="d7a33511-5d26-4719-a154-f436ab35b0cd" pod="openshift-must-gather-gx5nx/must-gather-zhcgp"
Jul 02 21:44:41.873396 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:41.873302    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckct8\" (UniqueName: \"kubernetes.io/projected/d7a33511-5d26-4719-a154-f436ab35b0cd-kube-api-access-ckct8\") pod \"d7a33511-5d26-4719-a154-f436ab35b0cd\" (UID: \"d7a33511-5d26-4719-a154-f436ab35b0cd\") "
Jul 02 21:44:41.873591 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:41.873438    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d7a33511-5d26-4719-a154-f436ab35b0cd-must-gather-output\") pod \"d7a33511-5d26-4719-a154-f436ab35b0cd\" (UID: \"d7a33511-5d26-4719-a154-f436ab35b0cd\") "
Jul 02 21:44:41.875537 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:41.875386    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a33511-5d26-4719-a154-f436ab35b0cd-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "d7a33511-5d26-4719-a154-f436ab35b0cd" (UID: "d7a33511-5d26-4719-a154-f436ab35b0cd"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:44:41.892576 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:41.892527    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a33511-5d26-4719-a154-f436ab35b0cd-kube-api-access-ckct8" (OuterVolumeSpecName: "kube-api-access-ckct8") pod "d7a33511-5d26-4719-a154-f436ab35b0cd" (UID: "d7a33511-5d26-4719-a154-f436ab35b0cd"). InnerVolumeSpecName "kube-api-access-ckct8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:44:41.974387 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:41.974331    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckct8\" (UniqueName: \"kubernetes.io/projected/d7a33511-5d26-4719-a154-f436ab35b0cd-kube-api-access-ckct8\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:44:41.974387 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:41.974361    2575 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d7a33511-5d26-4719-a154-f436ab35b0cd-must-gather-output\") on node \"ip-10-0-135-227.ec2.internal\" DevicePath \"\""
Jul 02 21:44:42.013051 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:42.012966    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:44:42.077860 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:42.077821    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnrb6_240069b9-1019-4c78-9b12-e2a30f6baeb6/konnectivity-agent/0.log" limit=1024
Jul 02 21:44:42.475027 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:42.474990    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-gx5nx_must-gather-zhcgp_d7a33511-5d26-4719-a154-f436ab35b0cd/copy/0.log"
Jul 02 21:44:42.475674 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:42.475612    2575 generic.go:356] "Generic (PLEG): container finished" podID="d7a33511-5d26-4719-a154-f436ab35b0cd" containerID="d610440baebf4c716d7f0101f53b5634d03f421132a7e39ac6602e1e64fd6424" exitCode=143
Jul 02 21:44:42.475793 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:42.475712    2575 scope.go:117] "RemoveContainer" containerID="d610440baebf4c716d7f0101f53b5634d03f421132a7e39ac6602e1e64fd6424"
Jul 02 21:44:42.475963 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:42.475943    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gx5nx/must-gather-zhcgp"
Jul 02 21:44:42.481697 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:44:42.481642    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-zhcgp\" is forbidden: User \"system:node:ip-10-0-135-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-gx5nx\": no relationship found between node 'ip-10-0-135-227.ec2.internal' and this object" podUID="d7a33511-5d26-4719-a154-f436ab35b0cd" pod="openshift-must-gather-gx5nx/must-gather-zhcgp"
Jul 02 21:44:42.488592 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:42.488568    2575 scope.go:117] "RemoveContainer" containerID="8bf97432dfb3cab66b9fabf7329a6cdc5f42b7188c2315e67d01d3ab56217914"
Jul 02 21:44:42.504454 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:44:42.501809    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-zhcgp\" is forbidden: User \"system:node:ip-10-0-135-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-gx5nx\": no relationship found between node 'ip-10-0-135-227.ec2.internal' and this object" podUID="d7a33511-5d26-4719-a154-f436ab35b0cd" pod="openshift-must-gather-gx5nx/must-gather-zhcgp"
Jul 02 21:44:42.514632 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:42.514331    2575 scope.go:117] "RemoveContainer" containerID="d610440baebf4c716d7f0101f53b5634d03f421132a7e39ac6602e1e64fd6424"
Jul 02 21:44:42.515084 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:44:42.514854    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d610440baebf4c716d7f0101f53b5634d03f421132a7e39ac6602e1e64fd6424\": container with ID starting with d610440baebf4c716d7f0101f53b5634d03f421132a7e39ac6602e1e64fd6424 not found: ID does not exist" containerID="d610440baebf4c716d7f0101f53b5634d03f421132a7e39ac6602e1e64fd6424"
Jul 02 21:44:42.515084 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:42.514885    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d610440baebf4c716d7f0101f53b5634d03f421132a7e39ac6602e1e64fd6424"} err="failed to get container status \"d610440baebf4c716d7f0101f53b5634d03f421132a7e39ac6602e1e64fd6424\": rpc error: code = NotFound desc = could not find container \"d610440baebf4c716d7f0101f53b5634d03f421132a7e39ac6602e1e64fd6424\": container with ID starting with d610440baebf4c716d7f0101f53b5634d03f421132a7e39ac6602e1e64fd6424 not found: ID does not exist"
Jul 02 21:44:42.515084 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:42.514906    2575 scope.go:117] "RemoveContainer" containerID="8bf97432dfb3cab66b9fabf7329a6cdc5f42b7188c2315e67d01d3ab56217914"
Jul 02 21:44:42.515643 ip-10-0-135-227 kubenswrapper[2575]: E0702 21:44:42.515144    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bf97432dfb3cab66b9fabf7329a6cdc5f42b7188c2315e67d01d3ab56217914\": container with ID starting with 8bf97432dfb3cab66b9fabf7329a6cdc5f42b7188c2315e67d01d3ab56217914 not found: ID does not exist" containerID="8bf97432dfb3cab66b9fabf7329a6cdc5f42b7188c2315e67d01d3ab56217914"
Jul 02 21:44:42.515643 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:42.515541    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bf97432dfb3cab66b9fabf7329a6cdc5f42b7188c2315e67d01d3ab56217914"} err="failed to get container status \"8bf97432dfb3cab66b9fabf7329a6cdc5f42b7188c2315e67d01d3ab56217914\": rpc error: code = NotFound desc = could not find container \"8bf97432dfb3cab66b9fabf7329a6cdc5f42b7188c2315e67d01d3ab56217914\": container with ID starting with 8bf97432dfb3cab66b9fabf7329a6cdc5f42b7188c2315e67d01d3ab56217914 not found: ID does not exist"
Jul 02 21:44:43.016461 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:43.016422    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7a33511-5d26-4719-a154-f436ab35b0cd" path="/var/lib/kubelet/pods/d7a33511-5d26-4719-a154-f436ab35b0cd/volumes"
Jul 02 21:44:43.555154 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:43.555066    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-t6jtm_cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9/kube-state-metrics/0.log"
Jul 02 21:44:43.585448 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:43.585426    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-t6jtm_cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9/kube-rbac-proxy-main/0.log"
Jul 02 21:44:43.614480 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:43.614446    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-t6jtm_cd8858d5-77d8-4ad7-b5d3-f8bc32332bc9/kube-rbac-proxy-self/0.log"
Jul 02 21:44:43.649465 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:43.649431    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-88d98b7c6-w4njm_0ca3427a-2e3a-496a-891c-326ea3e54437/metrics-server/0.log"
Jul 02 21:44:43.902096 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:43.902067    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lpkx5_a1f133dc-484c-4d5a-a67e-9fd63dbf861a/node-exporter/0.log"
Jul 02 21:44:43.927514 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:43.927479    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lpkx5_a1f133dc-484c-4d5a-a67e-9fd63dbf861a/kube-rbac-proxy/0.log"
Jul 02 21:44:43.960637 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:43.960608    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lpkx5_a1f133dc-484c-4d5a-a67e-9fd63dbf861a/init-textfile/0.log"
Jul 02 21:44:44.000097 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:44.000064    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-scqvn_ab2d2af2-5cf1-4ef5-ba32-685951b3595b/kube-rbac-proxy-main/0.log"
Jul 02 21:44:44.027395 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:44.027359    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-scqvn_ab2d2af2-5cf1-4ef5-ba32-685951b3595b/kube-rbac-proxy-self/0.log"
Jul 02 21:44:44.051449 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:44.051423    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-scqvn_ab2d2af2-5cf1-4ef5-ba32-685951b3595b/openshift-state-metrics/0.log"
Jul 02 21:44:44.108520 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:44.108487    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4773d17b-6ca7-4009-902f-7575c1d4b201/prometheus/0.log"
Jul 02 21:44:44.128044 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:44.128005    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4773d17b-6ca7-4009-902f-7575c1d4b201/config-reloader/0.log"
Jul 02 21:44:44.159308 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:44.159226    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4773d17b-6ca7-4009-902f-7575c1d4b201/thanos-sidecar/0.log"
Jul 02 21:44:44.186285 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:44.186241    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4773d17b-6ca7-4009-902f-7575c1d4b201/kube-rbac-proxy-web/0.log"
Jul 02 21:44:44.213199 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:44.213137    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4773d17b-6ca7-4009-902f-7575c1d4b201/kube-rbac-proxy/0.log"
Jul 02 21:44:44.235346 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:44.235300    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4773d17b-6ca7-4009-902f-7575c1d4b201/kube-rbac-proxy-thanos/0.log"
Jul 02 21:44:44.258136 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:44.258073    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4773d17b-6ca7-4009-902f-7575c1d4b201/init-config-reloader/0.log"
Jul 02 21:44:44.343510 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:44.343480    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-9dcr6_ef4203cc-0ee5-40b2-935c-6d81bf5b5948/prometheus-operator-admission-webhook/0.log"
Jul 02 21:44:44.471689 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:44.471602    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bb85b44fd-l7bk2_f1ca83f6-4c9b-4317-8455-239386da3440/thanos-query/0.log"
Jul 02 21:44:44.508510 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:44.508481    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bb85b44fd-l7bk2_f1ca83f6-4c9b-4317-8455-239386da3440/kube-rbac-proxy-web/0.log"
Jul 02 21:44:44.538025 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:44.537989    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bb85b44fd-l7bk2_f1ca83f6-4c9b-4317-8455-239386da3440/kube-rbac-proxy/0.log"
Jul 02 21:44:44.562228 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:44.562194    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bb85b44fd-l7bk2_f1ca83f6-4c9b-4317-8455-239386da3440/prom-label-proxy/0.log"
Jul 02 21:44:44.588622 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:44.588592    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bb85b44fd-l7bk2_f1ca83f6-4c9b-4317-8455-239386da3440/kube-rbac-proxy-rules/0.log"
Jul 02 21:44:44.614820 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:44.614793    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bb85b44fd-l7bk2_f1ca83f6-4c9b-4317-8455-239386da3440/kube-rbac-proxy-metrics/0.log"
Jul 02 21:44:45.912843 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:45.912811    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-ld997_48ccaac6-b95b-4f34-8fc1-be27faf2b2fc/networking-console-plugin/0.log"
Jul 02 21:44:47.399503 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.399472    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-7tvx2_91aa7825-8fff-45f4-9a55-bef98c760952/volume-data-source-validator/0.log"
Jul 02 21:44:47.537810 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.529505    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt"]
Jul 02 21:44:47.537810 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.530290    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7a33511-5d26-4719-a154-f436ab35b0cd" containerName="copy"
Jul 02 21:44:47.537810 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.530308    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7a33511-5d26-4719-a154-f436ab35b0cd" containerName="copy"
Jul 02 21:44:47.537810 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.530333    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7a33511-5d26-4719-a154-f436ab35b0cd" containerName="gather"
Jul 02 21:44:47.537810 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.530346    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7a33511-5d26-4719-a154-f436ab35b0cd" containerName="gather"
Jul 02 21:44:47.537810 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.530519    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7a33511-5d26-4719-a154-f436ab35b0cd" containerName="copy"
Jul 02 21:44:47.537810 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.530543    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7a33511-5d26-4719-a154-f436ab35b0cd" containerName="gather"
Jul 02 21:44:47.537810 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.536512    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt"
Jul 02 21:44:47.541825 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.541799    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt"]
Jul 02 21:44:47.632866 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.632809    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce-podres\") pod \"perf-node-gather-daemonset-qmnvt\" (UID: \"d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce\") " pod="openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt"
Jul 02 21:44:47.633065 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.632928    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w9lh\" (UniqueName: \"kubernetes.io/projected/d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce-kube-api-access-7w9lh\") pod \"perf-node-gather-daemonset-qmnvt\" (UID: \"d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce\") " pod="openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt"
Jul 02 21:44:47.633065 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.632983    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce-proc\") pod \"perf-node-gather-daemonset-qmnvt\" (UID: \"d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce\") " pod="openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt"
Jul 02 21:44:47.633065 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.633045    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce-sys\") pod \"perf-node-gather-daemonset-qmnvt\" (UID: \"d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce\") " pod="openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt"
Jul 02 21:44:47.633278 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.633066    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce-lib-modules\") pod \"perf-node-gather-daemonset-qmnvt\" (UID: \"d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce\") " pod="openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt"
Jul 02 21:44:47.734258 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.734148    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce-podres\") pod \"perf-node-gather-daemonset-qmnvt\" (UID: \"d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce\") " pod="openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt"
Jul 02 21:44:47.734417 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.734254    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w9lh\" (UniqueName: \"kubernetes.io/projected/d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce-kube-api-access-7w9lh\") pod \"perf-node-gather-daemonset-qmnvt\" (UID: \"d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce\") " pod="openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt"
Jul 02 21:44:47.734417 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.734314    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce-proc\") pod \"perf-node-gather-daemonset-qmnvt\" (UID: \"d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce\") " pod="openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt"
Jul 02 21:44:47.734417 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.734349    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce-podres\") pod \"perf-node-gather-daemonset-qmnvt\" (UID: \"d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce\") " pod="openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt"
Jul 02 21:44:47.734417 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.734377    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce-sys\") pod \"perf-node-gather-daemonset-qmnvt\" (UID: \"d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce\") " pod="openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt"
Jul 02 21:44:47.734417 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.734411    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce-lib-modules\") pod \"perf-node-gather-daemonset-qmnvt\" (UID: \"d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce\") " pod="openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt"
Jul 02 21:44:47.734685 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.734438    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce-proc\") pod \"perf-node-gather-daemonset-qmnvt\" (UID: \"d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce\") " pod="openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt"
Jul 02 21:44:47.734685 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.734485    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce-sys\") pod \"perf-node-gather-daemonset-qmnvt\" (UID: \"d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce\") " pod="openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt"
Jul 02 21:44:47.734685 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.734610    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce-lib-modules\") pod \"perf-node-gather-daemonset-qmnvt\" (UID: \"d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce\") " pod="openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt"
Jul 02 21:44:47.742728 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.742705    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w9lh\" (UniqueName: \"kubernetes.io/projected/d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce-kube-api-access-7w9lh\") pod \"perf-node-gather-daemonset-qmnvt\" (UID: \"d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce\") " pod="openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt"
Jul 02 21:44:47.851567 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:47.851524    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt"
Jul 02 21:44:48.032527 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:48.031795    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt"]
Jul 02 21:44:48.334107 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:48.334031    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7vjmt_c5609d59-a06f-4db8-b564-a0ff92bd58b3/dns/0.log"
Jul 02 21:44:48.356779 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:48.356748    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7vjmt_c5609d59-a06f-4db8-b564-a0ff92bd58b3/kube-rbac-proxy/0.log"
Jul 02 21:44:48.499379 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:48.499341    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k4r2h_a0c7410f-98e2-4331-b24b-6eeab7f99401/dns-node-resolver/0.log"
Jul 02 21:44:48.513220 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:48.513192    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt" event={"ID":"d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce","Type":"ContainerStarted","Data":"78cb753816adac8459efbaa2aa4f30719b7aeb77f8ba5b1de991e55e9d398e32"}
Jul 02 21:44:48.513386 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:48.513255    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt"
Jul 02 21:44:48.513386 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:48.513278    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt" event={"ID":"d236d8fb-fedb-4aa1-ad3f-aaed9ca25dce","Type":"ContainerStarted","Data":"2631929531f783fb408a874e9cc2c831b72e3c9225e2195a32b84785ac610f41"}
Jul 02 21:44:48.530001 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:48.529948    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt" podStartSLOduration=1.52993285 podStartE2EDuration="1.52993285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:44:48.527686993 +0000 UTC m=+1560.154188871" watchObservedRunningTime="2026-07-02 21:44:48.52993285 +0000 UTC m=+1560.156434751"
Jul 02 21:44:48.990094 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:48.990073    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9zmmn_83c0ff63-726d-4bb4-b4f6-0810ef1af2b2/node-ca/0.log"
Jul 02 21:44:50.228961 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:50.228934    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4xxrj_0c7a0d2a-4e86-46aa-9190-a0ce20694c85/serve-healthcheck-canary/0.log"
Jul 02 21:44:50.794526 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:50.794499    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pvf5d_06a47112-ec12-4baa-bd99-91b2d599254b/kube-rbac-proxy/0.log"
Jul 02 21:44:50.816457 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:50.816422    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pvf5d_06a47112-ec12-4baa-bd99-91b2d599254b/exporter/0.log"
Jul 02 21:44:50.836506 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:50.836483    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pvf5d_06a47112-ec12-4baa-bd99-91b2d599254b/extractor/0.log"
Jul 02 21:44:53.073410 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:53.073382    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-jmz4j_264e233b-c7a9-4bea-ae95-ccb7cc00bde6/s3-init/0.log"
Jul 02 21:44:53.090262 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:53.090234    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-jmz4j_264e233b-c7a9-4bea-ae95-ccb7cc00bde6/download-hf-model/0.log"
Jul 02 21:44:54.530065 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:54.530031    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-thbg4/perf-node-gather-daemonset-qmnvt"
Jul 02 21:44:57.521097 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:57.521071    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-pbcv9_78977169-1717-4b34-b8ac-7f4562f6be64/kube-storage-version-migrator-operator/1.log"
Jul 02 21:44:57.521883 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:57.521855    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-pbcv9_78977169-1717-4b34-b8ac-7f4562f6be64/kube-storage-version-migrator-operator/0.log"
Jul 02 21:44:58.495187 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:58.495147    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rfbk7_4b7e06c9-44c9-486b-8483-d71693ccc112/kube-multus-additional-cni-plugins/0.log"
Jul 02 21:44:58.516752 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:58.516724    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rfbk7_4b7e06c9-44c9-486b-8483-d71693ccc112/egress-router-binary-copy/0.log"
Jul 02 21:44:58.540952 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:58.540927    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rfbk7_4b7e06c9-44c9-486b-8483-d71693ccc112/cni-plugins/0.log"
Jul 02 21:44:58.582307 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:58.582282    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rfbk7_4b7e06c9-44c9-486b-8483-d71693ccc112/bond-cni-plugin/0.log"
Jul 02 21:44:58.603854 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:58.603826    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rfbk7_4b7e06c9-44c9-486b-8483-d71693ccc112/routeoverride-cni/0.log"
Jul 02 21:44:58.624813 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:58.624788    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rfbk7_4b7e06c9-44c9-486b-8483-d71693ccc112/whereabouts-cni-bincopy/0.log"
Jul 02 21:44:58.648829 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:58.648802    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rfbk7_4b7e06c9-44c9-486b-8483-d71693ccc112/whereabouts-cni/0.log"
Jul 02 21:44:58.909804 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:58.909767    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-w9fcp_dc228c55-194a-429b-a1dd-75001e9b6750/kube-multus/0.log"
Jul 02 21:44:59.028396 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:59.028361    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pxcm5_28a7f381-3db6-43e5-9690-d409f048dcf2/network-metrics-daemon/0.log"
Jul 02 21:44:59.048641 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:59.048617    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pxcm5_28a7f381-3db6-43e5-9690-d409f048dcf2/kube-rbac-proxy/0.log"
Jul 02 21:44:59.739226 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:59.739150    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2725k_bf253477-5a39-4648-9b18-4282286ab281/ovn-controller/0.log"
Jul 02 21:44:59.756152 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:59.756133    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2725k_bf253477-5a39-4648-9b18-4282286ab281/ovn-acl-logging/0.log"
Jul 02 21:44:59.780810 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:59.780772    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2725k_bf253477-5a39-4648-9b18-4282286ab281/kube-rbac-proxy-node/0.log"
Jul 02 21:44:59.799737 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:59.799711    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2725k_bf253477-5a39-4648-9b18-4282286ab281/kube-rbac-proxy-ovn-metrics/0.log"
Jul 02 21:44:59.817331 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:59.817311    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2725k_bf253477-5a39-4648-9b18-4282286ab281/northd/0.log"
Jul 02 21:44:59.840027 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:59.840007    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2725k_bf253477-5a39-4648-9b18-4282286ab281/nbdb/0.log"
Jul 02 21:44:59.861134 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:59.861101    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2725k_bf253477-5a39-4648-9b18-4282286ab281/sbdb/0.log"
Jul 02 21:44:59.905959 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:44:59.905938    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2725k_bf253477-5a39-4648-9b18-4282286ab281/ovnkube-controller/0.log"
Jul 02 21:45:01.295628 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:45:01.295604    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-b9tf7_4076e581-5038-4786-9565-c72be1618034/network-check-target-container/0.log"
Jul 02 21:45:02.195652 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:45:02.195621    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-qkjh4_3dbcf436-a3ce-467a-b081-2cfbc20d3af5/iptables-alerter/0.log"
Jul 02 21:45:02.852267 ip-10-0-135-227 kubenswrapper[2575]: I0702 21:45:02.852236    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-bb2r6_3ba76650-ff51-4f61-bbe0-cee05f1b91c6/tuned/0.log"