Sep 05 02:16:00.870195 ip-10-0-130-182 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 05 02:16:00.870206 ip-10-0-130-182 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 05 02:16:00.870216 ip-10-0-130-182 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 05 02:16:00.870415 ip-10-0-130-182 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 05 02:16:10.948453 ip-10-0-130-182 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 05 02:16:10.948475 ip-10-0-130-182 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 8913499a401f4edfaa24228ef42d28c6 --
Sep 05 02:17:47.256489 ip-10-0-130-182 systemd[1]: Starting Kubernetes Kubelet...
Sep 05 02:17:47.651140 ip-10-0-130-182 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:17:47.651140 ip-10-0-130-182 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 05 02:17:47.651140 ip-10-0-130-182 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:17:47.651140 ip-10-0-130-182 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 05 02:17:47.651140 ip-10-0-130-182 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:17:47.652259 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.652165    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 05 02:17:47.656737 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656724    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:47.656737 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656737    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:47.656800 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656740    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:47.656800 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656746    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:47.656800 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656749    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:47.656800 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656753    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:47.656800 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656756    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:47.656800 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656759    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:47.656800 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656761    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:47.656800 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656764    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:47.656800 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656767    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:47.656800 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656770    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:47.656800 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656772    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:47.656800 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656775    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:47.656800 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656778    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:47.656800 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656780    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:47.656800 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656783    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:47.656800 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656786    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:47.656800 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656789    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:47.656800 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656792    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:47.656800 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656795    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:47.656800 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656798    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:47.657323 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656800    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:47.657323 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656803    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:47.657323 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656806    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:47.657323 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656808    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:47.657323 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656811    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:47.657323 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656814    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:47.657323 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656817    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:47.657323 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656820    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:47.657323 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656822    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:47.657323 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656825    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:47.657323 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656828    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:47.657323 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656830    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:47.657323 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656833    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:47.657323 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656835    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:47.657323 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656838    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:47.657323 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656841    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:47.657323 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656843    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:47.657323 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656846    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:47.657323 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656848    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:47.657323 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656851    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:47.657794 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656853    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:47.657794 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656856    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:47.657794 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656858    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:47.657794 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656861    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:47.657794 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656863    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:47.657794 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656866    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:47.657794 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656868    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:47.657794 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656871    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:47.657794 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656874    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:47.657794 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656877    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:47.657794 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656879    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:47.657794 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656882    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:47.657794 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656885    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:47.657794 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656888    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:47.657794 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656891    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:47.657794 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656895    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:47.657794 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656898    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:47.657794 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656900    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:47.657794 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656903    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:47.657794 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656905    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:47.658301 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656908    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:47.658301 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656911    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:47.658301 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656914    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:47.658301 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656918    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:47.658301 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656922    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:47.658301 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656925    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:47.658301 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656928    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:47.658301 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656931    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:47.658301 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656934    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:47.658301 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656937    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:47.658301 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656940    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:47.658301 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656942    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:47.658301 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656946    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:47.658301 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656949    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:47.658301 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656952    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:47.658301 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656954    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:47.658301 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656957    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:47.658301 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656960    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:47.658301 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656963    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:47.658301 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656965    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:47.659064 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656970    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:47.659064 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656974    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:47.659064 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656977    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:47.659064 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656980    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:47.659064 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656983    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:47.659064 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656988    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:47.659064 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656991    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:47.659064 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656994    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:47.659064 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.656997    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:47.659064 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657000    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:47.659064 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657002    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:47.659064 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657005    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:47.659064 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657008    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:47.659064 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657010    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:47.659064 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657013    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:47.659064 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657015    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:47.659064 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657018    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:47.659064 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657730    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:47.659064 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657737    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:47.659064 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657740    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:47.659661 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657743    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:47.659661 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657746    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:47.659661 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657749    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:47.659661 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657751    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:47.659661 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657755    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:47.659661 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657757    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:47.659661 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657760    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:47.659661 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657762    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:47.659661 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657765    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:47.659661 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657768    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:47.659661 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657771    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:47.659661 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657774    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:47.659661 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657777    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:47.659661 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657780    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:47.659661 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657783    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:47.659661 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657786    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:47.659661 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657789    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:47.659661 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657793    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:47.659661 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657796    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:47.659661 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657798    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:47.660221 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657801    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:47.660221 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657804    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:47.660221 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657807    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:47.660221 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657809    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:47.660221 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657812    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:47.660221 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657815    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:47.660221 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657818    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:47.660221 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657820    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:47.660221 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657823    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:47.660221 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657825    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:47.660221 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657828    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:47.660221 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657830    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:47.660221 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657833    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:47.660221 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657836    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:47.660221 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657838    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:47.660221 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657841    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:47.660221 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657843    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:47.660221 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657846    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:47.660221 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657848    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:47.660685 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657852    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:47.660685 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657855    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:47.660685 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657861    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:47.660685 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657864    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:47.660685 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657867    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:47.660685 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657869    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:47.660685 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657872    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:47.660685 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657875    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:47.660685 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657877    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:47.660685 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657880    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:47.660685 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657883    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:47.660685 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657885    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:47.660685 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657888    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:47.660685 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657891    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:47.660685 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657893    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:47.660685 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657896    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:47.660685 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657899    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:47.660685 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657901    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:47.660685 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657904    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:47.660685 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657906    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:47.661184 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657911    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:47.661184 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657915    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:47.661184 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657918    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:47.661184 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657921    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:47.661184 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657924    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:47.661184 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657927    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:47.661184 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657929    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:47.661184 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657932    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:47.661184 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657934    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:47.661184 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657937    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:47.661184 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657939    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:47.661184 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657942    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:47.661184 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657946    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:47.661184 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657949    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:47.661184 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657953    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:47.661184 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657956    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:47.661184 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657959    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:47.661184 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657962    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:47.661184 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657965    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:47.661664 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657968    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:47.661664 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657971    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:47.661664 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657973    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:47.661664 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657976    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:47.661664 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657979    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:47.661664 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657981    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:47.661664 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657983    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:47.661664 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657986    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:47.661664 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657989    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:47.661664 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657991    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:47.661664 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657994    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:47.661664 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657996    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:47.661664 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.657999    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:47.661664 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.658001    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:47.661664 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.658004    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:47.661664 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.658007    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:47.661664 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.658009    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:47.661664 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.658012    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:47.661664 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661028    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:47.661664 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661040    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:47.662150 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661044    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:47.662150 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661047    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:47.662150 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661050    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:47.662150 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661054    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:47.662150 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661057    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:47.662150 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661060    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:47.662150 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661064    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:47.662150 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661068    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:47.662150 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661071    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:47.662150 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661073    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:47.662150 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661076    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:47.662150 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661079    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:47.662150 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661082    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:47.662150 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661084    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:47.662150 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661087    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:47.662150 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661090    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:47.662150 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661093    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:47.662150 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661095    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:47.662150 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661098    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:47.662637 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661101    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:47.662637 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661104    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:47.662637 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661107    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:47.662637 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661110    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:47.662637 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661112    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:47.662637 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661115    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:47.662637 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661118    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:47.662637 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661120    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:47.662637 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661123    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:47.662637 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661126    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:47.662637 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661129    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:47.662637 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661132    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:47.662637 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661135    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:47.662637 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661138    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:47.662637 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661141    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:47.662637 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661144    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:47.662637 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661146    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:47.662637 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661149    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:47.662637 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661152    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:47.662637 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661155    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:47.663127 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661157    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:47.663127 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661160    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:47.663127 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661162    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:47.663127 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661165    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:47.663127 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661168    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:47.663127 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661187    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:47.663127 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661192    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:47.663127 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661195    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:47.663127 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661198    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:47.663127 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661201    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:47.663127 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661204    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:47.663127 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661206    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:47.663127 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661209    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:47.663127 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661212    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:47.663127 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661215    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:47.663127 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661217    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:47.663127 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661220    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:47.663127 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661222    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:47.663127 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661225    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:47.663605 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661228    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:47.663605 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661230    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:47.663605 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661233    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:47.663605 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661235    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:47.663605 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661238    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:47.663605 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661241    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:47.663605 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661243    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:47.663605 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661246    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:47.663605 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661249    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:47.663605 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661251    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:47.663605 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661254    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:47.663605 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661257    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:47.663605 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661259    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:47.663605 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661262    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:47.663605 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661264    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:47.663605 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661267    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:47.663605 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661269    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:47.663605 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661272    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:47.663605 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661277    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:47.664068 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661281    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:47.664068 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661284    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:47.664068 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661287    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:47.664068 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661290    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:47.664068 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661294    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:47.664068 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661297    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:47.664068 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661299    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:47.664068 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661302    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:47.664068 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661305    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:47.664068 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661308    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:47.664068 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661312    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:47.664068 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661315    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:47.664068 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661318    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:47.664068 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661321    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:47.664068 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661324    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:47.664068 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661327    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:47.664068 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661330    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:47.664068 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661333    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:47.664068 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661335    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:47.664068 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.661338    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661415    2566 flags.go:64] FLAG: --address="0.0.0.0"
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661459    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661467    2566 flags.go:64] FLAG: --anonymous-auth="true"
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661472    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661476    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661479    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661484    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661488    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661491    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661494    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661497    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661500    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661503    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661506    2566 flags.go:64] FLAG: --cgroup-root=""
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661509    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661512    2566 flags.go:64] FLAG: --client-ca-file=""
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661515    2566 flags.go:64] FLAG: --cloud-provider="external"
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661519    2566 flags.go:64] FLAG: --cluster-dns="[]"
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661522    2566 flags.go:64] FLAG: --cluster-domain=""
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661526    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661529    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661532    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661536    2566 flags.go:64] FLAG: --container-log-max-files="5"
Sep 05 02:17:47.664566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661539    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661543    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661546    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661549    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661552    2566 flags.go:64] FLAG: --contention-profiling="false"
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661555    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661558    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661561    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661563    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661567    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661571    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661573    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661577    2566 flags.go:64] FLAG: --enable-load-reader="false"
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661580    2566 flags.go:64] FLAG: --enable-server="true"
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661583    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661587    2566 flags.go:64] FLAG: --event-burst="100"
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661591    2566 flags.go:64] FLAG: --event-qps="50"
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661594    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661597    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661600    2566 flags.go:64] FLAG: --eviction-hard=""
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661603    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661606    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661609    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661612    2566 flags.go:64] FLAG: --eviction-soft=""
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661615    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 05 02:17:47.665140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661618    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661621    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661624    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661627    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661630    2566 flags.go:64] FLAG: --fail-swap-on="true"
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661633    2566 flags.go:64] FLAG: --feature-gates=""
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661637    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661640    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661643    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661646    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661649    2566 flags.go:64] FLAG: --healthz-port="10248"
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661652    2566 flags.go:64] FLAG: --help="false"
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661655    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-130-182.ec2.internal"
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661658    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661661    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661664    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661667    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661671    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661674    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661680    2566 flags.go:64] FLAG: --image-service-endpoint=""
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661683    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661686    2566 flags.go:64] FLAG: --kube-api-burst="100"
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661689    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661692    2566 flags.go:64] FLAG: --kube-api-qps="50"
Sep 05 02:17:47.665814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661694    2566 flags.go:64] FLAG: --kube-reserved=""
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661697    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661700    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661703    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661706    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661709    2566 flags.go:64] FLAG: --lock-file=""
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661711    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661714    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661717    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661723    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661726    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661729    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661732    2566 flags.go:64] FLAG: --logging-format="text"
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661735    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661738    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661741    2566 flags.go:64] FLAG: --manifest-url=""
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661744    2566 flags.go:64] FLAG: --manifest-url-header=""
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661748    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661751    2566 flags.go:64] FLAG: --max-open-files="1000000"
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661755    2566 flags.go:64] FLAG: --max-pods="110"
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661758    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661760    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661763    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661766    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661769    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 05 02:17:47.666409 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661772    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661775    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661783    2566 flags.go:64] FLAG: --node-status-max-images="50"
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661786    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661789    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661792    2566 flags.go:64] FLAG: --pod-cidr=""
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661795    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661800    2566 flags.go:64] FLAG: --pod-manifest-path=""
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661802    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661806    2566 flags.go:64] FLAG: --pods-per-core="0"
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661809    2566 flags.go:64] FLAG: --port="10250"
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661812    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661815    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-053449fba1cc0ef07"
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661818    2566 flags.go:64] FLAG: --qos-reserved=""
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661821    2566 flags.go:64] FLAG: --read-only-port="10255"
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661824    2566 flags.go:64] FLAG: --register-node="true"
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661826    2566 flags.go:64] FLAG: --register-with-taints=""
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661830    2566 flags.go:64] FLAG: --registry-burst="10"
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661833    2566 flags.go:64] FLAG: --registry-qps="5"
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661836    2566 flags.go:64] FLAG: --reserved-cpus=""
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661839    2566 flags.go:64] FLAG: --reserved-memory=""
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661845    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661848    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661851    2566 flags.go:64] FLAG: --rotate-certificates="false"
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661854    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 05 02:17:47.667031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661857    2566 flags.go:64] FLAG: --runonce="false"
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661860    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661863    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661866    2566 flags.go:64] FLAG: --seccomp-default="false"
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661868    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661872    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661875    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661878    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661881    2566 flags.go:64] FLAG: --storage-driver-password="root"
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661884    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661887    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661890    2566 flags.go:64] FLAG: --storage-driver-user="root"
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661893    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661896    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661899    2566 flags.go:64] FLAG: --system-cgroups=""
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661902    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661908    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661910    2566 flags.go:64] FLAG: --tls-cert-file=""
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661913    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661917    2566 flags.go:64] FLAG: --tls-min-version=""
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661920    2566 flags.go:64] FLAG: --tls-private-key-file=""
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661922    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661925    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661928    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661931    2566 flags.go:64] FLAG: --v="2"
Sep 05 02:17:47.667661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661935    2566 flags.go:64] FLAG: --version="false"
Sep 05 02:17:47.668296 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661939    2566 flags.go:64] FLAG: --vmodule=""
Sep 05 02:17:47.668296 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661944    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 05 02:17:47.668296 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.661948    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 05 02:17:47.668296 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662062    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:47.668296 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662066    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:47.668296 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662070    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:47.668296 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662073    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:47.668296 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662076    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:47.668296 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662078    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:47.668296 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662081    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:47.668296 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662084    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:47.668296 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662087    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:47.668296 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662089    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:47.668296 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662092    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:47.668296 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662095    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:47.668296 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662097    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:47.668296 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662100    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:47.668296 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662103    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:47.668296 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662106    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:47.668296 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662110    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:47.668777 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662113    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:47.668777 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662116    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:47.668777 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662118    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:47.668777 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662121    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:47.668777 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662124    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:47.668777 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662126    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:47.668777 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662129    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:47.668777 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662132    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:47.668777 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662134    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:47.668777 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662136    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:47.668777 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662139    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:47.668777 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662142    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:47.668777 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662145    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:47.668777 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662147    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:47.668777 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662152    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:47.668777 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662155    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:47.668777 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662158    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:47.668777 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662160    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:47.668777 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662163    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:47.668777 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662166    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:47.669318 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662168    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:47.669318 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662185    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:47.669318 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662189    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:47.669318 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662192    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:47.669318 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662194    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:47.669318 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662198    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:47.669318 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662202    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:47.669318 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662206    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:47.669318 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662209    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:47.669318 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662212    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:47.669318 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662215    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:47.669318 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662218    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:47.669318 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662221    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:47.669318 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662224    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:47.669318 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662227    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:47.669318 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662229    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:47.669318 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662232    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:47.669318 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662234    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:47.669318 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662238    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:47.669788 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662242    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:47.669788 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662245    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:47.669788 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662247    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:47.669788 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662250    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:47.669788 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662252    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:47.669788 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662255    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:47.669788 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662257    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:47.669788 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662261    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:47.669788 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662265    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:47.669788 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662267    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:47.669788 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662270    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:47.669788 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662273    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:47.669788 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662275    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:47.669788 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662278    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:47.669788 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662280    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:47.669788 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662283    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:47.669788 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662285    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:47.669788 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662288    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:47.669788 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662290    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:47.669788 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662293    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:47.670300 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662297    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:47.670300 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662299    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:47.670300 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662302    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:47.670300 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662305    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:47.670300 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662307    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:47.670300 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662310    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:47.670300 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662313    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:47.670300 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662315    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:47.670300 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662318    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:47.670300 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662320    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:47.670300 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662323    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:47.670300 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662325    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:47.670300 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662328    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:47.670300 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662330    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:47.670300 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662333    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:47.670300 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662336    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:47.670300 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662338    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:47.670300 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662341    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:47.670300 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662343    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:47.670300 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662347    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:47.670787 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662349    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:47.670787 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662352    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:47.670787 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.662355    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:47.670787 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.662361    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:17:47.670787 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.669822    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 05 02:17:47.670787 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.669840    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 05 02:17:47.670787 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.670756    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 05 02:17:47.670787 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.670764    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 05 02:17:47.671003 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670821    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:47.671003 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670826    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:47.671003 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670830    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:47.671003 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670833    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:47.671003 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670836    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:47.671003 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670839    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:47.671003 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670842    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:47.671003 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670845    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:47.671003 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670848    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:47.671003 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670850    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:47.671003 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670853    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:47.671003 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670856    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:47.671003 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670859    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:47.671003 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670862    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:47.671003 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670864    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:47.671003 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670875    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:47.671003 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670878    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:47.671003 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670881    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:47.671003 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670884    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:47.671003 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670886    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:47.671511 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670889    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:47.671511 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670892    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:47.671511 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670894    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:47.671511 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670897    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:47.671511 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670900    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:47.671511 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670903    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:47.671511 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670905    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:47.671511 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670908    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:47.671511 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670911    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:47.671511 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670914    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:47.671511 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670917    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:47.671511 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670920    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:47.671511 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670923    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:47.671511 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670926    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:47.671511 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670929    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:47.671511 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670932    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:47.671511 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670934    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:47.671511 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670937    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:47.671511 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670939    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:47.671511 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670942    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:47.672063 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670945    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:47.672063 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670948    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:47.672063 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670950    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:47.672063 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670953    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:47.672063 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670955    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:47.672063 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670958    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:47.672063 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670960    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:47.672063 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670963    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:47.672063 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670966    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:47.672063 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670968    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:47.672063 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670970    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:47.672063 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670973    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:47.672063 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670976    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:47.672063 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670980    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:47.672063 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670983    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:47.672063 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670986    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:47.672063 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670989    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:47.672063 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670992    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:47.672063 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670995    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:47.672574 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.670998    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:47.672574 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671000    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:47.672574 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671003    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:47.672574 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671006    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:47.672574 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671009    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:47.672574 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671012    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:47.672574 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671015    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:47.672574 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671018    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:47.672574 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671020    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:47.672574 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671023    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:47.672574 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671026    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:47.672574 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671030    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:47.672574 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671033    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:47.672574 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671036    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:47.672574 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671039    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:47.672574 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671042    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:47.672574 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671045    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:47.672574 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671047    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:47.672574 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671050    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:47.673072 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671052    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:47.673072 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671055    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:47.673072 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671058    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:47.673072 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671060    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:47.673072 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671063    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:47.673072 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671066    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:47.673072 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671068    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:47.673072 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671071    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:47.673072 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671074    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:47.673072 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671076    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:47.673072 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671079    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:47.673072 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671081    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:47.673072 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671084    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:47.673072 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671086    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:47.673072 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671089    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:47.673072 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671092    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:47.673072 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671094    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:47.673072 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671099    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:47.673072 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671104    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:47.673560 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671107    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:47.673560 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671110    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:47.673560 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.671114    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:17:47.673560 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671244    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:47.673560 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671249    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:47.673560 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671252    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:47.673560 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671255    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:47.673560 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671258    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:47.673560 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671261    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:47.673560 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671265    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:47.673560 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671267    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:47.673560 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671270    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:47.673560 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671273    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:47.673560 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671276    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:47.673560 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671278    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:47.673930 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671281    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:47.673930 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671284    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:47.673930 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671286    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:47.673930 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671290    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:47.673930 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671292    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:47.673930 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671295    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:47.673930 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671297    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:47.673930 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671300    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:47.673930 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671303    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:47.673930 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671305    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:47.673930 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671308    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:47.673930 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671310    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:47.673930 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671313    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:47.673930 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671315    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:47.673930 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671318    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:47.673930 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671321    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:47.673930 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671324    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:47.673930 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671326    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:47.673930 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671329    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:47.673930 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671332    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:47.674424 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671334    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:47.674424 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671337    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:47.674424 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671340    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:47.674424 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671342    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:47.674424 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671345    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:47.674424 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671347    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:47.674424 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671350    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:47.674424 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671353    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:47.674424 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671355    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:47.674424 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671358    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:47.674424 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671360    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:47.674424 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671363    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:47.674424 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671365    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:47.674424 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671368    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:47.674424 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671370    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:47.674424 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671374    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:47.674424 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671378    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:47.674424 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671381    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:47.674424 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671385    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:47.674895 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671389    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:47.674895 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671392    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:47.674895 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671395    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:47.674895 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671398    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:47.674895 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671400    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:47.674895 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671422    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:47.674895 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671426    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:47.674895 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671429    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:47.674895 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671432    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:47.674895 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671435    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:47.674895 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671438    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:47.674895 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671441    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:47.674895 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671444    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:47.674895 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671447    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:47.674895 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671449    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:47.674895 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671452    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:47.674895 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671454    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:47.674895 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671457    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:47.674895 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671459    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:47.674895 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671462    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:47.675521 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671464    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:47.675521 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671467    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:47.675521 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671470    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:47.675521 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671473    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:47.675521 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671475    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:47.675521 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671478    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:47.675521 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671480    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:47.675521 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671483    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:47.675521 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671485    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:47.675521 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671488    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:47.675521 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671490    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:47.675521 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671493    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:47.675521 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671496    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:47.675521 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671498    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:47.675521 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671501    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:47.675521 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671503    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:47.675521 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671506    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:47.675521 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671508    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:47.675521 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671511    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:47.675521 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671513    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:47.676001 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671516    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:47.676001 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671519    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:47.676001 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671521    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:47.676001 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671525    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:47.676001 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671527    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:47.676001 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671530    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:47.676001 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671532    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:47.676001 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:47.671535    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:47.676001 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.671539    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:17:47.676001 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.671646    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 05 02:17:47.676001 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.674060    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 05 02:17:47.676001 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.675061    2566 server.go:1019] "Starting client certificate rotation"
Sep 05 02:17:47.676001 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.675164    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 05 02:17:47.676334 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.676080    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 05 02:17:47.677734 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.677721    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 05 02:17:47.681701 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.681671    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 05 02:17:47.692566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.692549    2566 log.go:25] "Validated CRI v1 runtime API"
Sep 05 02:17:47.695689 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.695674    2566 log.go:25] "Validated CRI v1 image API"
Sep 05 02:17:47.696581 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.696565    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 05 02:17:47.699029 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.699012    2566 fs.go:135] Filesystem UUIDs: map[51356b11-c0b0-407d-8120-dfddb51c4b9a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a0ee92bf-87bd-493c-9979-9fb2cc2c4bf7:/dev/nvme0n1p3]
Sep 05 02:17:47.699096 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.699028    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 05 02:17:47.702706 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.702683    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 05 02:17:47.705774 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.705661    2566 manager.go:216] Machine: {Timestamp:2026-09-05 02:17:47.704362339 +0000 UTC m=+0.347413741 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099251 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22c4de0112d331c8c288e14abbac3c SystemUUID:ec22c4de-0112-d331-c8c2-88e14abbac3c BootID:8913499a-401f-4edf-aa24-228ef42d28c6 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:cc:91:45:7d:09 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:cc:91:45:7d:09 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:92:c0:ff:0f:f3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 05 02:17:47.705774 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.705767    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 05 02:17:47.705925 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.705869    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 05 02:17:47.708615 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.708585    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 05 02:17:47.708741 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.708617    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-182.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 05 02:17:47.709500 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.709490    2566 topology_manager.go:138] "Creating topology manager with none policy"
Sep 05 02:17:47.709533 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.709503    2566 container_manager_linux.go:309] "Creating device plugin manager"
Sep 05 02:17:47.709533 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.709511    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 05 02:17:47.710395 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.710384    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 05 02:17:47.711557 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.711548    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 05 02:17:47.716679 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.716667    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 05 02:17:47.716782 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.716773    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 05 02:17:47.716815 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.716802    2566 kubelet.go:486] "Attempting to sync node with API server"
Sep 05 02:17:47.716815 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.716811    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 05 02:17:47.716877 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.716828    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 05 02:17:47.716877 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.716837    2566 kubelet.go:398] "Adding apiserver pod source"
Sep 05 02:17:47.716877 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.716846    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 05 02:17:47.717204 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.717192    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 05 02:17:47.717261 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.717211    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 05 02:17:47.719467 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.719451    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 05 02:17:47.722475 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.722462    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 05 02:17:47.722522 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.722492    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 05 02:17:47.724631 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.724616    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 05 02:17:47.724700 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.724629    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zdjjw"
Sep 05 02:17:47.724700 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.724637    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 05 02:17:47.724700 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.724645    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 05 02:17:47.724700 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.724650    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 05 02:17:47.724700 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.724659    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 05 02:17:47.724700 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.724664    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 05 02:17:47.724700 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.724678    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 05 02:17:47.724700 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.724683    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 05 02:17:47.724700 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.724689    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 05 02:17:47.724700 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.724694    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 05 02:17:47.724700 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.724703    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 05 02:17:47.724700 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.724709    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 05 02:17:47.725652 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.725642    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 05 02:17:47.725652 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.725652    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 05 02:17:47.726596 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:47.726571    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 05 02:17:47.726642 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:47.726576    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-182.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 05 02:17:47.727286 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.727272    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-182.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 05 02:17:47.729170 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.729158    2566 server.go:1268] "Started kubelet"
Sep 05 02:17:47.729262 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.729240    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 05 02:17:47.729319 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.729250    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 05 02:17:47.729319 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.729313    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 05 02:17:47.729852 ip-10-0-130-182 systemd[1]: Started Kubernetes Kubelet.
Sep 05 02:17:47.730907 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.730889    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 05 02:17:47.731026 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.731007    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zdjjw"
Sep 05 02:17:47.732448 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.732324    2566 server.go:310] "Adding debug handlers to kubelet server"
Sep 05 02:17:47.735577 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:47.735557    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 05 02:17:47.738021 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.738003    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 05 02:17:47.738100 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.738066    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 05 02:17:47.738160 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:47.738117    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Sep 05 02:17:47.738779 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.738764    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 05 02:17:47.738779 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.738781    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 05 02:17:47.738903 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.738765    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 05 02:17:47.738903 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.738842    2566 reconstruct.go:97] "Volume reconstruction finished"
Sep 05 02:17:47.738903 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.738850    2566 reconciler.go:29] "Reconciler: start to sync state"
Sep 05 02:17:47.738903 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.738888    2566 factory.go:55] Registering systemd factory
Sep 05 02:17:47.738903 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.738902    2566 factory.go:223] Registration of the systemd container factory successfully
Sep 05 02:17:47.739142 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.739130    2566 factory.go:153] Registering CRI-O factory
Sep 05 02:17:47.739194 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.739145    2566 factory.go:223] Registration of the crio container factory successfully
Sep 05 02:17:47.739227 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.739218    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 05 02:17:47.739257 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.739236    2566 factory.go:103] Registering Raw factory
Sep 05 02:17:47.739257 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.739247    2566 manager.go:1196] Started watching for new ooms in manager
Sep 05 02:17:47.739615 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.739603    2566 manager.go:349] Starting recovery of all containers
Sep 05 02:17:47.743584 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.743516    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:47.748162 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:47.748138    2566 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-182.ec2.internal\" not found" node="ip-10-0-130-182.ec2.internal"
Sep 05 02:17:47.753010 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.752994    2566 manager.go:354] Recovery completed
Sep 05 02:17:47.754163 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:47.754148    2566 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 05 02:17:47.757397 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.757299    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:47.759160 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.759140    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:47.759238 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.759185    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:47.759238 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.759199    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:47.759695 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.759675    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 05 02:17:47.759695 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.759692    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:47.759823 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.759694    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 05 02:17:47.759823 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.759724    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 05 02:17:47.760420 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.760402    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:47.760493 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.760434    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:47.760493 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.760448    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:47.761579 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.761567    2566 policy_none.go:49] "None policy: Start"
Sep 05 02:17:47.761623 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.761582    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 05 02:17:47.761623 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.761592    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 05 02:17:47.763342 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.763331    2566 policy_none.go:47] "Start"
Sep 05 02:17:47.784763 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.784749    2566 manager.go:337] "Starting Device Plugin manager"
Sep 05 02:17:47.784869 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:47.784779    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 05 02:17:47.784869 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.784788    2566 server.go:87] "Starting device plugin registration server"
Sep 05 02:17:47.785070 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.785050    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 05 02:17:47.785138 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.785069    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 05 02:17:47.785381 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.785338    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 05 02:17:47.785467 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.785409    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 05 02:17:47.785467 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.785419    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 05 02:17:47.786292 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:47.785678    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 05 02:17:47.786292 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:47.785725    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-182.ec2.internal\" not found"
Sep 05 02:17:47.877211 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.877170    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 05 02:17:47.879271 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.878270    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 05 02:17:47.879271 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.878317    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 05 02:17:47.879271 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.878340    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 05 02:17:47.879271 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:47.878426    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 05 02:17:47.880576 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.880555    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:47.885276 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.885262    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:47.886007 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.885993    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:47.886066 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.886018    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:47.886066 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.886028    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:47.886066 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.886047    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-182.ec2.internal"
Sep 05 02:17:47.894057 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.894041    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-182.ec2.internal"
Sep 05 02:17:47.894137 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:47.894075    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-182.ec2.internal\": node \"ip-10-0-130-182.ec2.internal\" not found"
Sep 05 02:17:47.905459 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:47.905394    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Sep 05 02:17:47.979307 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.979278    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal"]
Sep 05 02:17:47.979405 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.979351    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:47.980170 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.980147    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:47.980272 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.980196    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:47.980272 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.980213    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:47.982458 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.982444    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:47.982613 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.982599    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal"
Sep 05 02:17:47.982656 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.982628    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:47.983087 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.983075    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:47.983150 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.983095    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:47.983150 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.983114    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:47.983150 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.983140    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:47.983272 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.983158    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:47.983272 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.983167    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:47.985151 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.985136    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal"
Sep 05 02:17:47.985250 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.985159    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:47.985855 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.985838    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:47.985916 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.985867    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:47.985916 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:47.985878    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:48.005616 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:48.005592    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Sep 05 02:17:48.005719 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:48.005688    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-182.ec2.internal\" not found" node="ip-10-0-130-182.ec2.internal"
Sep 05 02:17:48.009937 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:48.009921    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-182.ec2.internal\" not found" node="ip-10-0-130-182.ec2.internal"
Sep 05 02:17:48.040705 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:48.040688    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d36b70f3f303d4cb4f8a7bad8d41d64d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-182.ec2.internal\" (UID: \"d36b70f3f303d4cb4f8a7bad8d41d64d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal"
Sep 05 02:17:48.040760 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:48.040710    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/568a76fd77addf714f0c72a696f95cb0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"568a76fd77addf714f0c72a696f95cb0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal"
Sep 05 02:17:48.040760 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:48.040732    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/568a76fd77addf714f0c72a696f95cb0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"568a76fd77addf714f0c72a696f95cb0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal"
Sep 05 02:17:48.106500 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:48.106478    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Sep 05 02:17:48.141221 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:48.141200    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d36b70f3f303d4cb4f8a7bad8d41d64d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-182.ec2.internal\" (UID: \"d36b70f3f303d4cb4f8a7bad8d41d64d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal"
Sep 05 02:17:48.141297 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:48.141224    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/568a76fd77addf714f0c72a696f95cb0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"568a76fd77addf714f0c72a696f95cb0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal"
Sep 05 02:17:48.141297 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:48.141241    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/568a76fd77addf714f0c72a696f95cb0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"568a76fd77addf714f0c72a696f95cb0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal"
Sep 05 02:17:48.141297 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:48.141265    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/568a76fd77addf714f0c72a696f95cb0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"568a76fd77addf714f0c72a696f95cb0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal"
Sep 05 02:17:48.141417 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:48.141301    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d36b70f3f303d4cb4f8a7bad8d41d64d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-182.ec2.internal\" (UID: \"d36b70f3f303d4cb4f8a7bad8d41d64d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal"
Sep 05 02:17:48.141417 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:48.141302    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/568a76fd77addf714f0c72a696f95cb0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"568a76fd77addf714f0c72a696f95cb0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal"
Sep 05 02:17:48.207489 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:48.207430    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Sep 05 02:17:48.308105 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:48.308073    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Sep 05 02:17:48.308105 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:48.308098    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal"
Sep 05 02:17:48.314773 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:48.313117    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal"
Sep 05 02:17:49.572807 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:49.572784    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Sep 05 02:17:49.673255 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:49.673205    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Sep 05 02:17:49.773763 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:49.773734    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Sep 05 02:17:49.838216 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:49.838183    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 05 02:17:49.838698 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:49.838348    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 05 02:17:49.838698 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:49.838361    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 05 02:17:49.874473 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:49.874451    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Sep 05 02:17:49.902063 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:49.902010    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-04 02:12:48 +0000 UTC" deadline="2028-02-11 22:52:28.420615549 +0000 UTC"
Sep 05 02:17:49.902063 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:49.902058    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12596h34m38.518561405s"
Sep 05 02:17:49.975527 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:49.975473    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Sep 05 02:17:50.027743 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.027720    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:50.075578 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:50.075553    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Sep 05 02:17:50.133641 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:50.133611    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36b70f3f303d4cb4f8a7bad8d41d64d.slice/crio-e36271a78b74ee9abd371b783bdbe3d7a79a4a6ff812f861a5fef40e260e4fdc WatchSource:0}: Error finding container e36271a78b74ee9abd371b783bdbe3d7a79a4a6ff812f861a5fef40e260e4fdc: Status 404 returned error can't find the container with id e36271a78b74ee9abd371b783bdbe3d7a79a4a6ff812f861a5fef40e260e4fdc
Sep 05 02:17:50.133861 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:50.133809    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568a76fd77addf714f0c72a696f95cb0.slice/crio-08fa3387c0a996bd9a3a18d60541d801ecedea398e848c36932bcf4a3788e059 WatchSource:0}: Error finding container 08fa3387c0a996bd9a3a18d60541d801ecedea398e848c36932bcf4a3788e059: Status 404 returned error can't find the container with id 08fa3387c0a996bd9a3a18d60541d801ecedea398e848c36932bcf4a3788e059
Sep 05 02:17:50.137360 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.137340    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:50.138358 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.138335    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:17:50.202074 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.202055    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal"
Sep 05 02:17:50.213485 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.213469    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 05 02:17:50.214380 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.214368    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal"
Sep 05 02:17:50.221431 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.221419    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 05 02:17:50.881770 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.881730    2566 apiserver.go:52] "Watching apiserver"
Sep 05 02:17:50.888820 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.888795    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 05 02:17:50.889312 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.889283    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-55qfp","openshift-network-operator/iptables-alerter-pp29d","openshift-ovn-kubernetes/ovnkube-node-7vnhf","kube-system/global-pull-secret-syncer-kj85s","kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal","openshift-cluster-node-tuning-operator/tuned-shqpm","openshift-image-registry/node-ca-vxccn","openshift-multus/multus-6wbh8","openshift-multus/multus-additional-cni-plugins-279ft","kube-system/konnectivity-agent-wxxsg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7","openshift-dns/node-resolver-b89gp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal","openshift-multus/network-metrics-daemon-262h2"]
Sep 05 02:17:50.891948 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.891887    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vxccn"
Sep 05 02:17:50.893937 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.893912    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:50.894632 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.894605    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 05 02:17:50.894738 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.894720    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 05 02:17:50.895067 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.894615    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mfwkn\""
Sep 05 02:17:50.895067 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.894894    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 05 02:17:50.896019 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.895999    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 05 02:17:50.896517 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.896500    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 05 02:17:50.896708 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.896691    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 05 02:17:50.896919 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.896904    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 05 02:17:50.897094 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.897079    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-l74gn\""
Sep 05 02:17:50.897878 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.897222    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:50.898809 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.898490    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:17:50.898809 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:50.898576    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj85s" podUID="37ca8378-c8ef-4342-b489-631cba71f8c5"
Sep 05 02:17:50.899216 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.899180    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 05 02:17:50.900015 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.899441    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 05 02:17:50.900015 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.899661    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 05 02:17:50.900015 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.899854    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 05 02:17:50.900015 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.899890    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 05 02:17:50.900244 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.900055    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 05 02:17:50.901067 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.900954    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vckmw\""
Sep 05 02:17:50.901193 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.901108    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:17:50.901193 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:50.901172    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55qfp" podUID="ac1f1f3d-3021-4a29-9162-77a512ca3a29"
Sep 05 02:17:50.903626 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.902789    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 05 02:17:50.903626 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.902799    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:50.905634 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.905136    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 05 02:17:50.905634 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.905357    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:50.907869 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.906735    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:17:50.907869 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.907052    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tqddg\""
Sep 05 02:17:50.908580 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.908559    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:17:50.908656 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:50.908633    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-262h2" podUID="e701e309-b8b1-47c5-927a-ce15959fafa1"
Sep 05 02:17:50.910469 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.909992    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4q4hj\""
Sep 05 02:17:50.911053 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.910855    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 05 02:17:50.911151 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.911134    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pp29d"
Sep 05 02:17:50.913089 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.913067    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 05 02:17:50.913665 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.913631    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 05 02:17:50.913888 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.913866    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c290fb8c-2fc3-4aff-863a-d8bf1ee41e01-serviceca\") pod \"node-ca-vxccn\" (UID: \"c290fb8c-2fc3-4aff-863a-d8bf1ee41e01\") " pod="openshift-image-registry/node-ca-vxccn"
Sep 05 02:17:50.913962 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.913904    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-multus-daemon-config\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:50.913962 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.913929    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d84gb\" (UniqueName: \"kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb\") pod \"network-check-target-55qfp\" (UID: \"ac1f1f3d-3021-4a29-9162-77a512ca3a29\") " pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:17:50.913962 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.913952    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-multus-cni-dir\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:50.914113 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.913977    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-etc-kubernetes\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:50.914113 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.913999    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-host-kubelet\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:50.914113 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914020    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eebe6f33-b592-4ee7-81e6-f3256af74fd4-etc-tuned\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:50.914113 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914042    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b864c4d-5153-4fed-a470-e6848614f613-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-279ft\" (UID: \"4b864c4d-5153-4fed-a470-e6848614f613\") " pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:50.914113 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914066    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f16c6647-7313-4ab5-b757-d95b95979d3c-ovn-node-metrics-cert\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:50.914113 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914089    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-etc-kubernetes\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:50.914113 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914111    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-host-run-ovn-kubernetes\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:50.914440 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914136    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/37ca8378-c8ef-4342-b489-631cba71f8c5-kubelet-config\") pod \"global-pull-secret-syncer-kj85s\" (UID: \"37ca8378-c8ef-4342-b489-631cba71f8c5\") " pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:17:50.914440 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914157    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-sys\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:50.914440 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914179    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-host-var-lib-kubelet\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:50.914440 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914202    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-systemd-units\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:50.914440 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914223    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-log-socket\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:50.914440 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914243    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-run-ovn\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:50.914440 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914266    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-etc-modprobe-d\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:50.914440 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914288    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-host-run-multus-certs\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:50.914440 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914310    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b864c4d-5153-4fed-a470-e6848614f613-cni-binary-copy\") pod \"multus-additional-cni-plugins-279ft\" (UID: \"4b864c4d-5153-4fed-a470-e6848614f613\") " pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:50.914440 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914334    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-multus-socket-dir-parent\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:50.914440 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914356    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-multus-conf-dir\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:50.914440 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914377    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-host-slash\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:50.914440 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914399    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-var-lib-openvswitch\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:50.914440 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914423    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-cni-binary-copy\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:50.915132 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914448    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-run-openvswitch\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:50.915132 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914470    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-var-lib-kubelet\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:50.915132 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914494    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f16c6647-7313-4ab5-b757-d95b95979d3c-ovnkube-config\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:50.915132 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b864c4d-5153-4fed-a470-e6848614f613-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-279ft\" (UID: \"4b864c4d-5153-4fed-a470-e6848614f613\") " pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:50.915132 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb5s4\" (UniqueName: \"kubernetes.io/projected/4b864c4d-5153-4fed-a470-e6848614f613-kube-api-access-vb5s4\") pod \"multus-additional-cni-plugins-279ft\" (UID: \"4b864c4d-5153-4fed-a470-e6848614f613\") " pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:50.915132 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914581    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-host-run-k8s-cni-cncf-io\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:50.915132 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914632    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-hostroot\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:50.915132 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914655    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzlxv\" (UniqueName: \"kubernetes.io/projected/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-kube-api-access-pzlxv\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:50.915132 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914678    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-etc-sysctl-conf\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:50.915132 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914720    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b864c4d-5153-4fed-a470-e6848614f613-system-cni-dir\") pod \"multus-additional-cni-plugins-279ft\" (UID: \"4b864c4d-5153-4fed-a470-e6848614f613\") " pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:50.915132 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914751    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-host-var-lib-cni-multus\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:50.915132 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914775    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-node-log\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:50.915132 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f16c6647-7313-4ab5-b757-d95b95979d3c-env-overrides\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:50.915132 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914825    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-etc-sysconfig\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:50.915132 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914862    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-etc-sysctl-d\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:50.915132 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b864c4d-5153-4fed-a470-e6848614f613-cnibin\") pod \"multus-additional-cni-plugins-279ft\" (UID: \"4b864c4d-5153-4fed-a470-e6848614f613\") " pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:50.915878 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-etc-openvswitch\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:50.915878 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914932    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f16c6647-7313-4ab5-b757-d95b95979d3c-ovnkube-script-lib\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:50.915878 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.914956    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret\") pod \"global-pull-secret-syncer-kj85s\" (UID: \"37ca8378-c8ef-4342-b489-631cba71f8c5\") " pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:17:50.915878 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.915174    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-host-run-netns\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:50.915878 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.915202    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-host-var-lib-cni-bin\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:50.915878 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.915226    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-etc-systemd\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:50.915878 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.915260    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-cnibin\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:50.915878 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.915282    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-host-cni-bin\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:50.915878 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.915314    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-run\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:50.915878 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.915335    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-host\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:50.915878 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.915355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eebe6f33-b592-4ee7-81e6-f3256af74fd4-tmp\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:50.915878 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.915378    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:50.915878 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.915405    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/37ca8378-c8ef-4342-b489-631cba71f8c5-dbus\") pod \"global-pull-secret-syncer-kj85s\" (UID: \"37ca8378-c8ef-4342-b489-631cba71f8c5\") " pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:17:50.915878 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.915569    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b864c4d-5153-4fed-a470-e6848614f613-tuning-conf-dir\") pod \"multus-additional-cni-plugins-279ft\" (UID: \"4b864c4d-5153-4fed-a470-e6848614f613\") " pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:50.915878 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.915606    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-run-systemd\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:50.915878 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.915634    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c290fb8c-2fc3-4aff-863a-d8bf1ee41e01-host\") pod \"node-ca-vxccn\" (UID: \"c290fb8c-2fc3-4aff-863a-d8bf1ee41e01\") " pod="openshift-image-registry/node-ca-vxccn"
Sep 05 02:17:50.915878 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.915658    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-host-run-netns\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:50.916582 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.915683    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9cfw\" (UniqueName: \"kubernetes.io/projected/f16c6647-7313-4ab5-b757-d95b95979d3c-kube-api-access-f9cfw\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:50.916582 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.915707    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-lib-modules\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:50.916582 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.915729    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jkjp\" (UniqueName: \"kubernetes.io/projected/eebe6f33-b592-4ee7-81e6-f3256af74fd4-kube-api-access-2jkjp\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:50.916582 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.915769    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr8kg\" (UniqueName: \"kubernetes.io/projected/c290fb8c-2fc3-4aff-863a-d8bf1ee41e01-kube-api-access-hr8kg\") pod \"node-ca-vxccn\" (UID: \"c290fb8c-2fc3-4aff-863a-d8bf1ee41e01\") " pod="openshift-image-registry/node-ca-vxccn"
Sep 05 02:17:50.916582 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.915803    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-system-cni-dir\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:50.916582 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.915853    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-host-cni-netd\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:50.916582 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.916019    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wxxsg"
Sep 05 02:17:50.916582 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.916153    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 05 02:17:50.916582 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.916340    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:17:50.916582 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.916393    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bxmp5\""
Sep 05 02:17:50.918457 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.918439    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 05 02:17:50.918811 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.918673    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 05 02:17:50.919464 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.919447    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zpjmv\""
Sep 05 02:17:50.920181 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.920153    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:50.921153 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.921134    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 05 02:17:50.922143 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.922119    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b89gp"
Sep 05 02:17:50.922239 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.922148    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 05 02:17:50.922401 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.922380    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 05 02:17:50.922607 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.922583    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 05 02:17:50.922727 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.922686    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2sxtz\""
Sep 05 02:17:50.924931 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.924172    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 05 02:17:50.924931 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.924494    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5pdfj\""
Sep 05 02:17:50.924931 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.924737    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 05 02:17:50.929849 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.929818    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:50.940697 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.940676    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dcb86"
Sep 05 02:17:50.948495 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:50.948475    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dcb86"
Sep 05 02:17:51.003282 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.003255    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 05 02:17:51.016651 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016256    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr8kg\" (UniqueName: \"kubernetes.io/projected/c290fb8c-2fc3-4aff-863a-d8bf1ee41e01-kube-api-access-hr8kg\") pod \"node-ca-vxccn\" (UID: \"c290fb8c-2fc3-4aff-863a-d8bf1ee41e01\") " pod="openshift-image-registry/node-ca-vxccn"
Sep 05 02:17:51.016651 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016290    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-system-cni-dir\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.016651 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016314    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-host-cni-netd\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.016651 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016343    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/974e8bfc-b802-4dfa-94c6-75a4331be5a5-konnectivity-ca\") pod \"konnectivity-agent-wxxsg\" (UID: \"974e8bfc-b802-4dfa-94c6-75a4331be5a5\") " pod="kube-system/konnectivity-agent-wxxsg"
Sep 05 02:17:51.016651 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016366    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c290fb8c-2fc3-4aff-863a-d8bf1ee41e01-serviceca\") pod \"node-ca-vxccn\" (UID: \"c290fb8c-2fc3-4aff-863a-d8bf1ee41e01\") " pod="openshift-image-registry/node-ca-vxccn"
Sep 05 02:17:51.016651 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016388    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24ff133a-c6d2-4cbc-8dd8-a1e662c05baf-hosts-file\") pod \"node-resolver-b89gp\" (UID: \"24ff133a-c6d2-4cbc-8dd8-a1e662c05baf\") " pod="openshift-dns/node-resolver-b89gp"
Sep 05 02:17:51.016651 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016412    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbn8d\" (UniqueName: \"kubernetes.io/projected/24ff133a-c6d2-4cbc-8dd8-a1e662c05baf-kube-api-access-sbn8d\") pod \"node-resolver-b89gp\" (UID: \"24ff133a-c6d2-4cbc-8dd8-a1e662c05baf\") " pod="openshift-dns/node-resolver-b89gp"
Sep 05 02:17:51.016651 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016436    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f534411f-681d-4622-8b46-747603a90cdc-device-dir\") pod \"aws-ebs-csi-driver-node-s96x7\" (UID: \"f534411f-681d-4622-8b46-747603a90cdc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:51.016651 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016459    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-multus-daemon-config\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.016651 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016484    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d84gb\" (UniqueName: \"kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb\") pod \"network-check-target-55qfp\" (UID: \"ac1f1f3d-3021-4a29-9162-77a512ca3a29\") " pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:17:51.016651 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016507    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-multus-cni-dir\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.016651 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016529    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-etc-kubernetes\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.016651 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016551    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-host-kubelet\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.016651 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016572    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eebe6f33-b592-4ee7-81e6-f3256af74fd4-etc-tuned\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.016651 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016594    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f534411f-681d-4622-8b46-747603a90cdc-etc-selinux\") pod \"aws-ebs-csi-driver-node-s96x7\" (UID: \"f534411f-681d-4622-8b46-747603a90cdc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:51.017473 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016662    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-host-cni-netd\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.017473 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016690    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b864c4d-5153-4fed-a470-e6848614f613-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-279ft\" (UID: \"4b864c4d-5153-4fed-a470-e6848614f613\") " pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:51.017473 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016718    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f16c6647-7313-4ab5-b757-d95b95979d3c-ovn-node-metrics-cert\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.017473 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016723    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-multus-cni-dir\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.017473 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-etc-kubernetes\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.017473 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016768    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-etc-kubernetes\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.017473 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016778    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-system-cni-dir\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.017473 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.016888    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-etc-kubernetes\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.017473 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.017032    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 05 02:17:51.017902 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.017501    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fsxj\" (UniqueName: \"kubernetes.io/projected/f534411f-681d-4622-8b46-747603a90cdc-kube-api-access-9fsxj\") pod \"aws-ebs-csi-driver-node-s96x7\" (UID: \"f534411f-681d-4622-8b46-747603a90cdc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:51.017902 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.017540    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-host-run-ovn-kubernetes\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.017902 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.017580    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c290fb8c-2fc3-4aff-863a-d8bf1ee41e01-serviceca\") pod \"node-ca-vxccn\" (UID: \"c290fb8c-2fc3-4aff-863a-d8bf1ee41e01\") " pod="openshift-image-registry/node-ca-vxccn"
Sep 05 02:17:51.017902 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.017609    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-host-run-ovn-kubernetes\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.017902 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.017610    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-host-kubelet\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.017902 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.017646    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/37ca8378-c8ef-4342-b489-631cba71f8c5-kubelet-config\") pod \"global-pull-secret-syncer-kj85s\" (UID: \"37ca8378-c8ef-4342-b489-631cba71f8c5\") " pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:17:51.017902 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.017693    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-multus-daemon-config\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.017902 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.017754    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-sys\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.019654 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.019626    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b864c4d-5153-4fed-a470-e6848614f613-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-279ft\" (UID: \"4b864c4d-5153-4fed-a470-e6848614f613\") " pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:51.019773 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.019701    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-sys\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.019773 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.019744    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f534411f-681d-4622-8b46-747603a90cdc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s96x7\" (UID: \"f534411f-681d-4622-8b46-747603a90cdc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:51.019906 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.019777    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-host-var-lib-kubelet\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.019906 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.019807    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-systemd-units\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.019906 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.019858    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-log-socket\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.019906 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.019886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6qkt\" (UniqueName: \"kubernetes.io/projected/e701e309-b8b1-47c5-927a-ce15959fafa1-kube-api-access-k6qkt\") pod \"network-metrics-daemon-262h2\" (UID: \"e701e309-b8b1-47c5-927a-ce15959fafa1\") " pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:17:51.020097 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.019920    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-run-ovn\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.020097 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.019949    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-etc-modprobe-d\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.020097 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.019978    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-host-run-multus-certs\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.020097 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020004    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b864c4d-5153-4fed-a470-e6848614f613-cni-binary-copy\") pod \"multus-additional-cni-plugins-279ft\" (UID: \"4b864c4d-5153-4fed-a470-e6848614f613\") " pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:51.020097 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020033    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-multus-socket-dir-parent\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.020097 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020060    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-multus-conf-dir\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.020097 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020088    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-host-slash\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.020358 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020115    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-var-lib-openvswitch\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.020358 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-cni-binary-copy\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.020358 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020171    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-run-openvswitch\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.020358 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020173    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-run-ovn\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.020358 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020253    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-multus-conf-dir\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.020358 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020290    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-etc-modprobe-d\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.020358 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020310    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-host-run-multus-certs\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.020358 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020334    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-var-lib-kubelet\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.020635 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020366    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f16c6647-7313-4ab5-b757-d95b95979d3c-ovnkube-config\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.020635 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020394    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ff133a-c6d2-4cbc-8dd8-a1e662c05baf-tmp-dir\") pod \"node-resolver-b89gp\" (UID: \"24ff133a-c6d2-4cbc-8dd8-a1e662c05baf\") " pod="openshift-dns/node-resolver-b89gp"
Sep 05 02:17:51.020635 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020424    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f534411f-681d-4622-8b46-747603a90cdc-registration-dir\") pod \"aws-ebs-csi-driver-node-s96x7\" (UID: \"f534411f-681d-4622-8b46-747603a90cdc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:51.020635 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020457    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b864c4d-5153-4fed-a470-e6848614f613-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-279ft\" (UID: \"4b864c4d-5153-4fed-a470-e6848614f613\") " pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:51.020635 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020490    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb5s4\" (UniqueName: \"kubernetes.io/projected/4b864c4d-5153-4fed-a470-e6848614f613-kube-api-access-vb5s4\") pod \"multus-additional-cni-plugins-279ft\" (UID: \"4b864c4d-5153-4fed-a470-e6848614f613\") " pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:51.020635 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020522    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-host-run-k8s-cni-cncf-io\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.020635 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020549    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-hostroot\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.020635 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020581    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzlxv\" (UniqueName: \"kubernetes.io/projected/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-kube-api-access-pzlxv\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.020635 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020605    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-multus-socket-dir-parent\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.020635 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020613    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-etc-sysctl-conf\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.021040 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020651    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b864c4d-5153-4fed-a470-e6848614f613-system-cni-dir\") pod \"multus-additional-cni-plugins-279ft\" (UID: \"4b864c4d-5153-4fed-a470-e6848614f613\") " pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:51.021040 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020680    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-host-var-lib-cni-multus\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.021040 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020711    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-node-log\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.021040 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020716    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-etc-sysctl-conf\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.021040 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020744    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f16c6647-7313-4ab5-b757-d95b95979d3c-env-overrides\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.021040 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020763    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-host-slash\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.021040 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-etc-sysconfig\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.021040 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020814    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-var-lib-openvswitch\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.021040 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020872    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-etc-sysctl-d\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.021040 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b864c4d-5153-4fed-a470-e6848614f613-cnibin\") pod \"multus-additional-cni-plugins-279ft\" (UID: \"4b864c4d-5153-4fed-a470-e6848614f613\") " pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:51.021040 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020935    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-etc-openvswitch\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.021040 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020967    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f16c6647-7313-4ab5-b757-d95b95979d3c-ovnkube-script-lib\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.021040 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.020994    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret\") pod \"global-pull-secret-syncer-kj85s\" (UID: \"37ca8378-c8ef-4342-b489-631cba71f8c5\") " pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:17:51.021040 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.021029    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f534411f-681d-4622-8b46-747603a90cdc-socket-dir\") pod \"aws-ebs-csi-driver-node-s96x7\" (UID: \"f534411f-681d-4622-8b46-747603a90cdc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:51.025366 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.025344    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-host-run-netns\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.025366 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.021295    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f16c6647-7313-4ab5-b757-d95b95979d3c-env-overrides\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.025496 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.021396    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-hostroot\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.025496 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.021442    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-log-socket\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.025496 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.021487    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-etc-sysconfig\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.025496 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.021346    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-run-openvswitch\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.025496 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.021524    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b864c4d-5153-4fed-a470-e6848614f613-system-cni-dir\") pod \"multus-additional-cni-plugins-279ft\" (UID: \"4b864c4d-5153-4fed-a470-e6848614f613\") " pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:51.025496 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.021589    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-node-log\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.025496 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.021630    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-var-lib-kubelet\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.025496 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.021699    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-cni-binary-copy\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.025496 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.021736    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eebe6f33-b592-4ee7-81e6-f3256af74fd4-etc-tuned\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.025496 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.025471    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-host-var-lib-cni-bin\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.025496 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.021755    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-host-var-lib-kubelet\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.025496 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.021795    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b864c4d-5153-4fed-a470-e6848614f613-cnibin\") pod \"multus-additional-cni-plugins-279ft\" (UID: \"4b864c4d-5153-4fed-a470-e6848614f613\") " pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:51.025496 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.021798    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-etc-openvswitch\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.026169 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.022164    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-etc-sysctl-d\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.026169 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.022076    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-systemd-units\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.026169 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.022184    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f16c6647-7313-4ab5-b757-d95b95979d3c-ovnkube-config\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.026169 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.022323    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f16c6647-7313-4ab5-b757-d95b95979d3c-ovnkube-script-lib\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.026169 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.022388    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-host-run-k8s-cni-cncf-io\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.026169 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.022576    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/37ca8378-c8ef-4342-b489-631cba71f8c5-kubelet-config\") pod \"global-pull-secret-syncer-kj85s\" (UID: \"37ca8378-c8ef-4342-b489-631cba71f8c5\") " pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:17:51.026169 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.022683    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b864c4d-5153-4fed-a470-e6848614f613-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-279ft\" (UID: \"4b864c4d-5153-4fed-a470-e6848614f613\") " pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:51.026169 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.022735    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b864c4d-5153-4fed-a470-e6848614f613-cni-binary-copy\") pod \"multus-additional-cni-plugins-279ft\" (UID: \"4b864c4d-5153-4fed-a470-e6848614f613\") " pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:51.026169 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.021556    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-host-var-lib-cni-multus\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.026169 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.025390    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-host-var-lib-cni-bin\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.026169 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.025457    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-host-run-netns\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.026169 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.025634    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-etc-systemd\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.026169 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:51.022246    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:51.026169 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.025663    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e81e0ea-1618-456d-b0f4-33c3e3f3e518-host-slash\") pod \"iptables-alerter-pp29d\" (UID: \"7e81e0ea-1618-456d-b0f4-33c3e3f3e518\") " pod="openshift-network-operator/iptables-alerter-pp29d"
Sep 05 02:17:51.026169 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.025689    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/974e8bfc-b802-4dfa-94c6-75a4331be5a5-agent-certs\") pod \"konnectivity-agent-wxxsg\" (UID: \"974e8bfc-b802-4dfa-94c6-75a4331be5a5\") " pod="kube-system/konnectivity-agent-wxxsg"
Sep 05 02:17:51.026169 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:51.025219    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:51.026169 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.025706    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-etc-systemd\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.026169 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.025714    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f534411f-681d-4622-8b46-747603a90cdc-sys-fs\") pod \"aws-ebs-csi-driver-node-s96x7\" (UID: \"f534411f-681d-4622-8b46-747603a90cdc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:51.027005 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:51.025731    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret podName:37ca8378-c8ef-4342-b489-631cba71f8c5 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:51.525709435 +0000 UTC m=+3.005103592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret") pod "global-pull-secret-syncer-kj85s" (UID: "37ca8378-c8ef-4342-b489-631cba71f8c5") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:51.027005 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:51.025730    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:51.027005 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:51.025755    2566 projected.go:196] Error preparing data for projected volume kube-api-access-d84gb for pod openshift-network-diagnostics/network-check-target-55qfp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:51.027005 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.025773    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-cnibin\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.027005 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.025798    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-host-cni-bin\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.027005 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.025845    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-run\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.027005 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:51.025868    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb podName:ac1f1f3d-3021-4a29-9162-77a512ca3a29 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:51.525851768 +0000 UTC m=+3.005245924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d84gb" (UniqueName: "kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb") pod "network-check-target-55qfp" (UID: "ac1f1f3d-3021-4a29-9162-77a512ca3a29") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:51.027005 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.025877    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-cnibin\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.027005 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.025894    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-run\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.027005 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.025908    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-host\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.027005 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.025941    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-host-cni-bin\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.027005 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.025942    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eebe6f33-b592-4ee7-81e6-f3256af74fd4-tmp\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.027005 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.025952    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-host\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.027005 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.025984    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.027005 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.026036    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/37ca8378-c8ef-4342-b489-631cba71f8c5-dbus\") pod \"global-pull-secret-syncer-kj85s\" (UID: \"37ca8378-c8ef-4342-b489-631cba71f8c5\") " pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:17:51.027005 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.026085    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b864c4d-5153-4fed-a470-e6848614f613-tuning-conf-dir\") pod \"multus-additional-cni-plugins-279ft\" (UID: \"4b864c4d-5153-4fed-a470-e6848614f613\") " pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:51.027788 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.026115    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-run-systemd\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.027788 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.026145    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs\") pod \"network-metrics-daemon-262h2\" (UID: \"e701e309-b8b1-47c5-927a-ce15959fafa1\") " pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:17:51.027788 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.026168    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c290fb8c-2fc3-4aff-863a-d8bf1ee41e01-host\") pod \"node-ca-vxccn\" (UID: \"c290fb8c-2fc3-4aff-863a-d8bf1ee41e01\") " pod="openshift-image-registry/node-ca-vxccn"
Sep 05 02:17:51.027788 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.026198    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-host-run-netns\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.027788 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.026227    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9cfw\" (UniqueName: \"kubernetes.io/projected/f16c6647-7313-4ab5-b757-d95b95979d3c-kube-api-access-f9cfw\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.027788 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.026256    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-lib-modules\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.027788 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.026279    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jkjp\" (UniqueName: \"kubernetes.io/projected/eebe6f33-b592-4ee7-81e6-f3256af74fd4-kube-api-access-2jkjp\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.027788 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.026309    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e81e0ea-1618-456d-b0f4-33c3e3f3e518-iptables-alerter-script\") pod \"iptables-alerter-pp29d\" (UID: \"7e81e0ea-1618-456d-b0f4-33c3e3f3e518\") " pod="openshift-network-operator/iptables-alerter-pp29d"
Sep 05 02:17:51.027788 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.026341    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd6fq\" (UniqueName: \"kubernetes.io/projected/7e81e0ea-1618-456d-b0f4-33c3e3f3e518-kube-api-access-zd6fq\") pod \"iptables-alerter-pp29d\" (UID: \"7e81e0ea-1618-456d-b0f4-33c3e3f3e518\") " pod="openshift-network-operator/iptables-alerter-pp29d"
Sep 05 02:17:51.027788 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.026387    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.027788 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.026595    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/37ca8378-c8ef-4342-b489-631cba71f8c5-dbus\") pod \"global-pull-secret-syncer-kj85s\" (UID: \"37ca8378-c8ef-4342-b489-631cba71f8c5\") " pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:17:51.027788 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.027000    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c290fb8c-2fc3-4aff-863a-d8bf1ee41e01-host\") pod \"node-ca-vxccn\" (UID: \"c290fb8c-2fc3-4aff-863a-d8bf1ee41e01\") " pod="openshift-image-registry/node-ca-vxccn"
Sep 05 02:17:51.027788 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.027112    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eebe6f33-b592-4ee7-81e6-f3256af74fd4-lib-modules\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.027788 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.027162    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-host-run-netns\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.027788 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.027547    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b864c4d-5153-4fed-a470-e6848614f613-tuning-conf-dir\") pod \"multus-additional-cni-plugins-279ft\" (UID: \"4b864c4d-5153-4fed-a470-e6848614f613\") " pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:51.027788 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.027720    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f16c6647-7313-4ab5-b757-d95b95979d3c-run-systemd\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.029472 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.028616    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr8kg\" (UniqueName: \"kubernetes.io/projected/c290fb8c-2fc3-4aff-863a-d8bf1ee41e01-kube-api-access-hr8kg\") pod \"node-ca-vxccn\" (UID: \"c290fb8c-2fc3-4aff-863a-d8bf1ee41e01\") " pod="openshift-image-registry/node-ca-vxccn"
Sep 05 02:17:51.029472 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.028783    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eebe6f33-b592-4ee7-81e6-f3256af74fd4-tmp\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.029700 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.029632    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzlxv\" (UniqueName: \"kubernetes.io/projected/5fa3cda0-ec69-4fe0-b459-119a5d7d4d42-kube-api-access-pzlxv\") pod \"multus-6wbh8\" (UID: \"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42\") " pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.030591 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.030549    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb5s4\" (UniqueName: \"kubernetes.io/projected/4b864c4d-5153-4fed-a470-e6848614f613-kube-api-access-vb5s4\") pod \"multus-additional-cni-plugins-279ft\" (UID: \"4b864c4d-5153-4fed-a470-e6848614f613\") " pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:51.030710 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.030689    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f16c6647-7313-4ab5-b757-d95b95979d3c-ovn-node-metrics-cert\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.039454 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.039407    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9cfw\" (UniqueName: \"kubernetes.io/projected/f16c6647-7313-4ab5-b757-d95b95979d3c-kube-api-access-f9cfw\") pod \"ovnkube-node-7vnhf\" (UID: \"f16c6647-7313-4ab5-b757-d95b95979d3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.043011 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.042980    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jkjp\" (UniqueName: \"kubernetes.io/projected/eebe6f33-b592-4ee7-81e6-f3256af74fd4-kube-api-access-2jkjp\") pod \"tuned-shqpm\" (UID: \"eebe6f33-b592-4ee7-81e6-f3256af74fd4\") " pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.047382 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.046969    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" event={"ID":"d36b70f3f303d4cb4f8a7bad8d41d64d","Type":"ContainerStarted","Data":"e36271a78b74ee9abd371b783bdbe3d7a79a4a6ff812f861a5fef40e260e4fdc"}
Sep 05 02:17:51.049495 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.049440    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" event={"ID":"568a76fd77addf714f0c72a696f95cb0","Type":"ContainerStarted","Data":"08fa3387c0a996bd9a3a18d60541d801ecedea398e848c36932bcf4a3788e059"}
Sep 05 02:17:51.127546 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.127442    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ff133a-c6d2-4cbc-8dd8-a1e662c05baf-tmp-dir\") pod \"node-resolver-b89gp\" (UID: \"24ff133a-c6d2-4cbc-8dd8-a1e662c05baf\") " pod="openshift-dns/node-resolver-b89gp"
Sep 05 02:17:51.127546 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.127485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f534411f-681d-4622-8b46-747603a90cdc-registration-dir\") pod \"aws-ebs-csi-driver-node-s96x7\" (UID: \"f534411f-681d-4622-8b46-747603a90cdc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:51.127546 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.127534    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f534411f-681d-4622-8b46-747603a90cdc-socket-dir\") pod \"aws-ebs-csi-driver-node-s96x7\" (UID: \"f534411f-681d-4622-8b46-747603a90cdc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:51.127810 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.127556    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e81e0ea-1618-456d-b0f4-33c3e3f3e518-host-slash\") pod \"iptables-alerter-pp29d\" (UID: \"7e81e0ea-1618-456d-b0f4-33c3e3f3e518\") " pod="openshift-network-operator/iptables-alerter-pp29d"
Sep 05 02:17:51.127810 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.127595    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/974e8bfc-b802-4dfa-94c6-75a4331be5a5-agent-certs\") pod \"konnectivity-agent-wxxsg\" (UID: \"974e8bfc-b802-4dfa-94c6-75a4331be5a5\") " pod="kube-system/konnectivity-agent-wxxsg"
Sep 05 02:17:51.127810 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.127615    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f534411f-681d-4622-8b46-747603a90cdc-sys-fs\") pod \"aws-ebs-csi-driver-node-s96x7\" (UID: \"f534411f-681d-4622-8b46-747603a90cdc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:51.127810 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.127629    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f534411f-681d-4622-8b46-747603a90cdc-registration-dir\") pod \"aws-ebs-csi-driver-node-s96x7\" (UID: \"f534411f-681d-4622-8b46-747603a90cdc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:51.127810 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.127642    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs\") pod \"network-metrics-daemon-262h2\" (UID: \"e701e309-b8b1-47c5-927a-ce15959fafa1\") " pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:17:51.127810 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.127669    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e81e0ea-1618-456d-b0f4-33c3e3f3e518-iptables-alerter-script\") pod \"iptables-alerter-pp29d\" (UID: \"7e81e0ea-1618-456d-b0f4-33c3e3f3e518\") " pod="openshift-network-operator/iptables-alerter-pp29d"
Sep 05 02:17:51.127810 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.127681    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e81e0ea-1618-456d-b0f4-33c3e3f3e518-host-slash\") pod \"iptables-alerter-pp29d\" (UID: \"7e81e0ea-1618-456d-b0f4-33c3e3f3e518\") " pod="openshift-network-operator/iptables-alerter-pp29d"
Sep 05 02:17:51.127810 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.127690    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd6fq\" (UniqueName: \"kubernetes.io/projected/7e81e0ea-1618-456d-b0f4-33c3e3f3e518-kube-api-access-zd6fq\") pod \"iptables-alerter-pp29d\" (UID: \"7e81e0ea-1618-456d-b0f4-33c3e3f3e518\") " pod="openshift-network-operator/iptables-alerter-pp29d"
Sep 05 02:17:51.127810 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.127712    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f534411f-681d-4622-8b46-747603a90cdc-sys-fs\") pod \"aws-ebs-csi-driver-node-s96x7\" (UID: \"f534411f-681d-4622-8b46-747603a90cdc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:51.127810 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.127698    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f534411f-681d-4622-8b46-747603a90cdc-socket-dir\") pod \"aws-ebs-csi-driver-node-s96x7\" (UID: \"f534411f-681d-4622-8b46-747603a90cdc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:51.127810 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.127751    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/974e8bfc-b802-4dfa-94c6-75a4331be5a5-konnectivity-ca\") pod \"konnectivity-agent-wxxsg\" (UID: \"974e8bfc-b802-4dfa-94c6-75a4331be5a5\") " pod="kube-system/konnectivity-agent-wxxsg"
Sep 05 02:17:51.127810 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.127779    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24ff133a-c6d2-4cbc-8dd8-a1e662c05baf-hosts-file\") pod \"node-resolver-b89gp\" (UID: \"24ff133a-c6d2-4cbc-8dd8-a1e662c05baf\") " pod="openshift-dns/node-resolver-b89gp"
Sep 05 02:17:51.127810 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.127805    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbn8d\" (UniqueName: \"kubernetes.io/projected/24ff133a-c6d2-4cbc-8dd8-a1e662c05baf-kube-api-access-sbn8d\") pod \"node-resolver-b89gp\" (UID: \"24ff133a-c6d2-4cbc-8dd8-a1e662c05baf\") " pod="openshift-dns/node-resolver-b89gp"
Sep 05 02:17:51.128435 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.127828    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f534411f-681d-4622-8b46-747603a90cdc-device-dir\") pod \"aws-ebs-csi-driver-node-s96x7\" (UID: \"f534411f-681d-4622-8b46-747603a90cdc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:51.128435 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.127917    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24ff133a-c6d2-4cbc-8dd8-a1e662c05baf-hosts-file\") pod \"node-resolver-b89gp\" (UID: \"24ff133a-c6d2-4cbc-8dd8-a1e662c05baf\") " pod="openshift-dns/node-resolver-b89gp"
Sep 05 02:17:51.128435 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.127958    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f534411f-681d-4622-8b46-747603a90cdc-etc-selinux\") pod \"aws-ebs-csi-driver-node-s96x7\" (UID: \"f534411f-681d-4622-8b46-747603a90cdc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:51.128435 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.127990    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fsxj\" (UniqueName: \"kubernetes.io/projected/f534411f-681d-4622-8b46-747603a90cdc-kube-api-access-9fsxj\") pod \"aws-ebs-csi-driver-node-s96x7\" (UID: \"f534411f-681d-4622-8b46-747603a90cdc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:51.128435 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:51.128012    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:51.128435 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.128018    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f534411f-681d-4622-8b46-747603a90cdc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s96x7\" (UID: \"f534411f-681d-4622-8b46-747603a90cdc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:51.128435 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.128046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6qkt\" (UniqueName: \"kubernetes.io/projected/e701e309-b8b1-47c5-927a-ce15959fafa1-kube-api-access-k6qkt\") pod \"network-metrics-daemon-262h2\" (UID: \"e701e309-b8b1-47c5-927a-ce15959fafa1\") " pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:17:51.128435 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:51.128079    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs podName:e701e309-b8b1-47c5-927a-ce15959fafa1 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:51.628059973 +0000 UTC m=+3.107454114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs") pod "network-metrics-daemon-262h2" (UID: "e701e309-b8b1-47c5-927a-ce15959fafa1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:51.128435 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.128112    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f534411f-681d-4622-8b46-747603a90cdc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s96x7\" (UID: \"f534411f-681d-4622-8b46-747603a90cdc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:51.128435 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.128145    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f534411f-681d-4622-8b46-747603a90cdc-etc-selinux\") pod \"aws-ebs-csi-driver-node-s96x7\" (UID: \"f534411f-681d-4622-8b46-747603a90cdc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:51.128435 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.128209    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f534411f-681d-4622-8b46-747603a90cdc-device-dir\") pod \"aws-ebs-csi-driver-node-s96x7\" (UID: \"f534411f-681d-4622-8b46-747603a90cdc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:51.128435 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.128222    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ff133a-c6d2-4cbc-8dd8-a1e662c05baf-tmp-dir\") pod \"node-resolver-b89gp\" (UID: \"24ff133a-c6d2-4cbc-8dd8-a1e662c05baf\") " pod="openshift-dns/node-resolver-b89gp"
Sep 05 02:17:51.129010 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.128504    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e81e0ea-1618-456d-b0f4-33c3e3f3e518-iptables-alerter-script\") pod \"iptables-alerter-pp29d\" (UID: \"7e81e0ea-1618-456d-b0f4-33c3e3f3e518\") " pod="openshift-network-operator/iptables-alerter-pp29d"
Sep 05 02:17:51.129010 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.128557    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/974e8bfc-b802-4dfa-94c6-75a4331be5a5-konnectivity-ca\") pod \"konnectivity-agent-wxxsg\" (UID: \"974e8bfc-b802-4dfa-94c6-75a4331be5a5\") " pod="kube-system/konnectivity-agent-wxxsg"
Sep 05 02:17:51.130699 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.130678    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/974e8bfc-b802-4dfa-94c6-75a4331be5a5-agent-certs\") pod \"konnectivity-agent-wxxsg\" (UID: \"974e8bfc-b802-4dfa-94c6-75a4331be5a5\") " pod="kube-system/konnectivity-agent-wxxsg"
Sep 05 02:17:51.136397 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.136342    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fsxj\" (UniqueName: \"kubernetes.io/projected/f534411f-681d-4622-8b46-747603a90cdc-kube-api-access-9fsxj\") pod \"aws-ebs-csi-driver-node-s96x7\" (UID: \"f534411f-681d-4622-8b46-747603a90cdc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:51.136504 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.136448    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd6fq\" (UniqueName: \"kubernetes.io/projected/7e81e0ea-1618-456d-b0f4-33c3e3f3e518-kube-api-access-zd6fq\") pod \"iptables-alerter-pp29d\" (UID: \"7e81e0ea-1618-456d-b0f4-33c3e3f3e518\") " pod="openshift-network-operator/iptables-alerter-pp29d"
Sep 05 02:17:51.136879 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.136857    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbn8d\" (UniqueName: \"kubernetes.io/projected/24ff133a-c6d2-4cbc-8dd8-a1e662c05baf-kube-api-access-sbn8d\") pod \"node-resolver-b89gp\" (UID: \"24ff133a-c6d2-4cbc-8dd8-a1e662c05baf\") " pod="openshift-dns/node-resolver-b89gp"
Sep 05 02:17:51.137057 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.137039    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6qkt\" (UniqueName: \"kubernetes.io/projected/e701e309-b8b1-47c5-927a-ce15959fafa1-kube-api-access-k6qkt\") pod \"network-metrics-daemon-262h2\" (UID: \"e701e309-b8b1-47c5-927a-ce15959fafa1\") " pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:17:51.214272 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.214212    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vxccn"
Sep 05 02:17:51.223721 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.223686    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6wbh8"
Sep 05 02:17:51.232414 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.232394    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:17:51.240258 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.240237    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-shqpm"
Sep 05 02:17:51.254907 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.254886    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-279ft"
Sep 05 02:17:51.258350 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.258327    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:51.264122 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.264102    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pp29d"
Sep 05 02:17:51.271759 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.271740    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wxxsg"
Sep 05 02:17:51.278448 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.278429    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7"
Sep 05 02:17:51.284967 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.284949    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b89gp"
Sep 05 02:17:51.529994 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.529901    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret\") pod \"global-pull-secret-syncer-kj85s\" (UID: \"37ca8378-c8ef-4342-b489-631cba71f8c5\") " pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:17:51.529994 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.529972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d84gb\" (UniqueName: \"kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb\") pod \"network-check-target-55qfp\" (UID: \"ac1f1f3d-3021-4a29-9162-77a512ca3a29\") " pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:17:51.530212 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:51.530069    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:51.530212 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:51.530105    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:51.530212 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:51.530131    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:51.530212 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:51.530145    2566 projected.go:196] Error preparing data for projected volume kube-api-access-d84gb for pod openshift-network-diagnostics/network-check-target-55qfp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:51.530212 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:51.530163    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret podName:37ca8378-c8ef-4342-b489-631cba71f8c5 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:52.530144979 +0000 UTC m=+4.009539127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret") pod "global-pull-secret-syncer-kj85s" (UID: "37ca8378-c8ef-4342-b489-631cba71f8c5") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:51.530212 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:51.530194    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb podName:ac1f1f3d-3021-4a29-9162-77a512ca3a29 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:52.530182322 +0000 UTC m=+4.009576459 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d84gb" (UniqueName: "kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb") pod "network-check-target-55qfp" (UID: "ac1f1f3d-3021-4a29-9162-77a512ca3a29") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:51.631134 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.631097    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs\") pod \"network-metrics-daemon-262h2\" (UID: \"e701e309-b8b1-47c5-927a-ce15959fafa1\") " pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:17:51.631323 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:51.631261    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:51.631369 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:51.631324    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs podName:e701e309-b8b1-47c5-927a-ce15959fafa1 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:52.631310094 +0000 UTC m=+4.110704234 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs") pod "network-metrics-daemon-262h2" (UID: "e701e309-b8b1-47c5-927a-ce15959fafa1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:51.952150 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.952064    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 02:12:50 +0000 UTC" deadline="2028-02-11 11:02:30.861518209 +0000 UTC"
Sep 05 02:17:51.952150 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:51.952102    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12584h44m38.90942206s"
Sep 05 02:17:52.014038 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:52.013999    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc290fb8c_2fc3_4aff_863a_d8bf1ee41e01.slice/crio-44c1bc9f97e9efba23cfab9e5e2072c6153779f1f869ddfe0ce819d251683008 WatchSource:0}: Error finding container 44c1bc9f97e9efba23cfab9e5e2072c6153779f1f869ddfe0ce819d251683008: Status 404 returned error can't find the container with id 44c1bc9f97e9efba23cfab9e5e2072c6153779f1f869ddfe0ce819d251683008
Sep 05 02:17:52.015215 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:52.015188    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e81e0ea_1618_456d_b0f4_33c3e3f3e518.slice/crio-031a357e836c6e2931fa3b2bd6cd40c813e58bdda20132add561d5fbf8fa6fec WatchSource:0}: Error finding container 031a357e836c6e2931fa3b2bd6cd40c813e58bdda20132add561d5fbf8fa6fec: Status 404 returned error can't find the container with id 031a357e836c6e2931fa3b2bd6cd40c813e58bdda20132add561d5fbf8fa6fec
Sep 05 02:17:52.016021 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:52.015985    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa3cda0_ec69_4fe0_b459_119a5d7d4d42.slice/crio-e0050b8698998e2d599cb16fbd073c5122394004802b37a435bb438da751b8e2 WatchSource:0}: Error finding container e0050b8698998e2d599cb16fbd073c5122394004802b37a435bb438da751b8e2: Status 404 returned error can't find the container with id e0050b8698998e2d599cb16fbd073c5122394004802b37a435bb438da751b8e2
Sep 05 02:17:52.020016 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:52.019987    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16c6647_7313_4ab5_b757_d95b95979d3c.slice/crio-1de7d9c74af2552c1bf41933fac9d53bce080eb5e396a44bcc0838d36768ba80 WatchSource:0}: Error finding container 1de7d9c74af2552c1bf41933fac9d53bce080eb5e396a44bcc0838d36768ba80: Status 404 returned error can't find the container with id 1de7d9c74af2552c1bf41933fac9d53bce080eb5e396a44bcc0838d36768ba80
Sep 05 02:17:52.040519 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:52.040482    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeebe6f33_b592_4ee7_81e6_f3256af74fd4.slice/crio-6ace176b6300dda7b9560511ad6492a4348f678b39335d85869fcba31abb9202 WatchSource:0}: Error finding container 6ace176b6300dda7b9560511ad6492a4348f678b39335d85869fcba31abb9202: Status 404 returned error can't find the container with id 6ace176b6300dda7b9560511ad6492a4348f678b39335d85869fcba31abb9202
Sep 05 02:17:52.041344 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:52.041320    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ff133a_c6d2_4cbc_8dd8_a1e662c05baf.slice/crio-6b64e33cb25f18497fc6093466bfd43d602252764982274ebcafa9311755af4f WatchSource:0}: Error finding container 6b64e33cb25f18497fc6093466bfd43d602252764982274ebcafa9311755af4f: Status 404 returned error can't find the container with id 6b64e33cb25f18497fc6093466bfd43d602252764982274ebcafa9311755af4f
Sep 05 02:17:52.042284 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:52.042205    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:17:52.042391 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:52.042281    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf534411f_681d_4622_8b46_747603a90cdc.slice/crio-a1108797851ee5859f7b8fcc2c65b50b679b026db67a92633c0803381e163d8e WatchSource:0}: Error finding container a1108797851ee5859f7b8fcc2c65b50b679b026db67a92633c0803381e163d8e: Status 404 returned error can't find the container with id a1108797851ee5859f7b8fcc2c65b50b679b026db67a92633c0803381e163d8e
Sep 05 02:17:52.042391 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:52.042208    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:17:52.042391 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:52.042315    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj85s" podUID="37ca8378-c8ef-4342-b489-631cba71f8c5"
Sep 05 02:17:52.042545 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:52.042406    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-262h2" podUID="e701e309-b8b1-47c5-927a-ce15959fafa1"
Sep 05 02:17:52.043114 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:52.043015    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod974e8bfc_b802_4dfa_94c6_75a4331be5a5.slice/crio-1273dd9141b3502e6bad1ce77818b1fc2cd0eca9f82aa1173715e931b01c88ca WatchSource:0}: Error finding container 1273dd9141b3502e6bad1ce77818b1fc2cd0eca9f82aa1173715e931b01c88ca: Status 404 returned error can't find the container with id 1273dd9141b3502e6bad1ce77818b1fc2cd0eca9f82aa1173715e931b01c88ca
Sep 05 02:17:52.044632 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:17:52.044573    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b864c4d_5153_4fed_a470_e6848614f613.slice/crio-d8c7ecba9ee6fdae573323ab1ba01e883ba4e719012b67cbb44eb4b61e5647f3 WatchSource:0}: Error finding container d8c7ecba9ee6fdae573323ab1ba01e883ba4e719012b67cbb44eb4b61e5647f3: Status 404 returned error can't find the container with id d8c7ecba9ee6fdae573323ab1ba01e883ba4e719012b67cbb44eb4b61e5647f3
Sep 05 02:17:52.051736 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:52.051713    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-shqpm" event={"ID":"eebe6f33-b592-4ee7-81e6-f3256af74fd4","Type":"ContainerStarted","Data":"6ace176b6300dda7b9560511ad6492a4348f678b39335d85869fcba31abb9202"}
Sep 05 02:17:52.053447 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:52.053423    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wxxsg" event={"ID":"974e8bfc-b802-4dfa-94c6-75a4331be5a5","Type":"ContainerStarted","Data":"1273dd9141b3502e6bad1ce77818b1fc2cd0eca9f82aa1173715e931b01c88ca"}
Sep 05 02:17:52.056074 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:52.055995    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7" event={"ID":"f534411f-681d-4622-8b46-747603a90cdc","Type":"ContainerStarted","Data":"a1108797851ee5859f7b8fcc2c65b50b679b026db67a92633c0803381e163d8e"}
Sep 05 02:17:52.058416 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:52.058392    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b89gp" event={"ID":"24ff133a-c6d2-4cbc-8dd8-a1e662c05baf","Type":"ContainerStarted","Data":"6b64e33cb25f18497fc6093466bfd43d602252764982274ebcafa9311755af4f"}
Sep 05 02:17:52.063031 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:52.062985    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf" event={"ID":"f16c6647-7313-4ab5-b757-d95b95979d3c","Type":"ContainerStarted","Data":"1de7d9c74af2552c1bf41933fac9d53bce080eb5e396a44bcc0838d36768ba80"}
Sep 05 02:17:52.064178 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:52.064151    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-279ft" event={"ID":"4b864c4d-5153-4fed-a470-e6848614f613","Type":"ContainerStarted","Data":"d8c7ecba9ee6fdae573323ab1ba01e883ba4e719012b67cbb44eb4b61e5647f3"}
Sep 05 02:17:52.065789 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:52.065584    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6wbh8" event={"ID":"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42","Type":"ContainerStarted","Data":"e0050b8698998e2d599cb16fbd073c5122394004802b37a435bb438da751b8e2"}
Sep 05 02:17:52.067171 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:52.067150    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pp29d" event={"ID":"7e81e0ea-1618-456d-b0f4-33c3e3f3e518","Type":"ContainerStarted","Data":"031a357e836c6e2931fa3b2bd6cd40c813e58bdda20132add561d5fbf8fa6fec"}
Sep 05 02:17:52.068160 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:52.068138    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vxccn" event={"ID":"c290fb8c-2fc3-4aff-863a-d8bf1ee41e01","Type":"ContainerStarted","Data":"44c1bc9f97e9efba23cfab9e5e2072c6153779f1f869ddfe0ce819d251683008"}
Sep 05 02:17:52.537289 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:52.537184    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret\") pod \"global-pull-secret-syncer-kj85s\" (UID: \"37ca8378-c8ef-4342-b489-631cba71f8c5\") " pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:17:52.537289 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:52.537260    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d84gb\" (UniqueName: \"kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb\") pod \"network-check-target-55qfp\" (UID: \"ac1f1f3d-3021-4a29-9162-77a512ca3a29\") " pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:17:52.537509 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:52.537407    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:52.537509 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:52.537425    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:52.537509 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:52.537439    2566 projected.go:196] Error preparing data for projected volume kube-api-access-d84gb for pod openshift-network-diagnostics/network-check-target-55qfp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:52.537509 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:52.537497    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb podName:ac1f1f3d-3021-4a29-9162-77a512ca3a29 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:54.537478018 +0000 UTC m=+6.016872169 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d84gb" (UniqueName: "kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb") pod "network-check-target-55qfp" (UID: "ac1f1f3d-3021-4a29-9162-77a512ca3a29") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:52.538012 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:52.537914    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:52.538012 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:52.537969    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret podName:37ca8378-c8ef-4342-b489-631cba71f8c5 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:54.537953426 +0000 UTC m=+6.017347562 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret") pod "global-pull-secret-syncer-kj85s" (UID: "37ca8378-c8ef-4342-b489-631cba71f8c5") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:52.637632 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:52.637600    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs\") pod \"network-metrics-daemon-262h2\" (UID: \"e701e309-b8b1-47c5-927a-ce15959fafa1\") " pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:17:52.637792 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:52.637776    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:52.637897 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:52.637851    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs podName:e701e309-b8b1-47c5-927a-ce15959fafa1 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:54.637817016 +0000 UTC m=+6.117211165 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs") pod "network-metrics-daemon-262h2" (UID: "e701e309-b8b1-47c5-927a-ce15959fafa1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:52.953063 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:52.952924    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 02:12:50 +0000 UTC" deadline="2028-02-11 17:10:42.001702092 +0000 UTC"
Sep 05 02:17:52.953063 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:52.952985    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12590h52m49.048720873s"
Sep 05 02:17:53.042494 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:53.042454    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:17:53.042670 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:53.042593    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55qfp" podUID="ac1f1f3d-3021-4a29-9162-77a512ca3a29"
Sep 05 02:17:53.087947 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:53.087868    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" event={"ID":"d36b70f3f303d4cb4f8a7bad8d41d64d","Type":"ContainerStarted","Data":"fb3e4fead25623eb9a50fda077d6c9cc92b53bbc6d28c4e04211f844245dbccc"}
Sep 05 02:17:53.108491 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:53.108451    2566 generic.go:356] "Generic (PLEG): container finished" podID="568a76fd77addf714f0c72a696f95cb0" containerID="e48e0cf0933c1e78d2f3f10b9795d16ecd6b0f8a227c16e38ddc80597182af21" exitCode=0
Sep 05 02:17:53.108661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:53.108500    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" event={"ID":"568a76fd77addf714f0c72a696f95cb0","Type":"ContainerDied","Data":"e48e0cf0933c1e78d2f3f10b9795d16ecd6b0f8a227c16e38ddc80597182af21"}
Sep 05 02:17:53.133092 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:53.132105    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" podStartSLOduration=3.132081176 podStartE2EDuration="3.132081176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:17:53.103110041 +0000 UTC m=+4.582504200" watchObservedRunningTime="2026-09-05 02:17:53.132081176 +0000 UTC m=+4.611475338"
Sep 05 02:17:54.042706 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:54.042673    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:17:54.043401 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:54.042816    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-262h2" podUID="e701e309-b8b1-47c5-927a-ce15959fafa1"
Sep 05 02:17:54.043401 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:54.043209    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:17:54.043401 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:54.043300    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj85s" podUID="37ca8378-c8ef-4342-b489-631cba71f8c5"
Sep 05 02:17:54.141043 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:54.140956    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" event={"ID":"568a76fd77addf714f0c72a696f95cb0","Type":"ContainerStarted","Data":"76145dbbe6f4fc3f4a287ed8a2a7ccd952eeb5e5c28613602aff897b2ec31ea0"}
Sep 05 02:17:54.165050 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:54.164992    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" podStartSLOduration=4.164969238 podStartE2EDuration="4.164969238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:17:54.164170889 +0000 UTC m=+5.643565047" watchObservedRunningTime="2026-09-05 02:17:54.164969238 +0000 UTC m=+5.644363397"
Sep 05 02:17:54.553737 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:54.553649    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d84gb\" (UniqueName: \"kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb\") pod \"network-check-target-55qfp\" (UID: \"ac1f1f3d-3021-4a29-9162-77a512ca3a29\") " pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:17:54.553737 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:54.553718    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret\") pod \"global-pull-secret-syncer-kj85s\" (UID: \"37ca8378-c8ef-4342-b489-631cba71f8c5\") " pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:17:54.553978 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:54.553864    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:54.553978 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:54.553920    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret podName:37ca8378-c8ef-4342-b489-631cba71f8c5 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:58.553903987 +0000 UTC m=+10.033298130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret") pod "global-pull-secret-syncer-kj85s" (UID: "37ca8378-c8ef-4342-b489-631cba71f8c5") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:54.554342 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:54.554316    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:54.554342 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:54.554342    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:54.554495 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:54.554354    2566 projected.go:196] Error preparing data for projected volume kube-api-access-d84gb for pod openshift-network-diagnostics/network-check-target-55qfp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:54.554495 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:54.554395    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb podName:ac1f1f3d-3021-4a29-9162-77a512ca3a29 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:58.55438077 +0000 UTC m=+10.033774919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d84gb" (UniqueName: "kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb") pod "network-check-target-55qfp" (UID: "ac1f1f3d-3021-4a29-9162-77a512ca3a29") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:54.655090 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:54.655026    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs\") pod \"network-metrics-daemon-262h2\" (UID: \"e701e309-b8b1-47c5-927a-ce15959fafa1\") " pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:17:54.655278 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:54.655241    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:54.655335 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:54.655304    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs podName:e701e309-b8b1-47c5-927a-ce15959fafa1 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:58.655285766 +0000 UTC m=+10.134679917 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs") pod "network-metrics-daemon-262h2" (UID: "e701e309-b8b1-47c5-927a-ce15959fafa1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:55.042763 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:55.042683    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:17:55.043222 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:55.042889    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55qfp" podUID="ac1f1f3d-3021-4a29-9162-77a512ca3a29"
Sep 05 02:17:56.043173 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:56.042456    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:17:56.043173 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:56.042591    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj85s" podUID="37ca8378-c8ef-4342-b489-631cba71f8c5"
Sep 05 02:17:56.043173 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:56.043013    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:17:56.043173 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:56.043124    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-262h2" podUID="e701e309-b8b1-47c5-927a-ce15959fafa1"
Sep 05 02:17:57.044702 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:57.044673    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:17:57.045170 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:57.044789    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55qfp" podUID="ac1f1f3d-3021-4a29-9162-77a512ca3a29"
Sep 05 02:17:58.042587 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:58.042487    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:17:58.042754 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:58.042609    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj85s" podUID="37ca8378-c8ef-4342-b489-631cba71f8c5"
Sep 05 02:17:58.042983 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:58.042960    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:17:58.043086 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:58.043054    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-262h2" podUID="e701e309-b8b1-47c5-927a-ce15959fafa1"
Sep 05 02:17:58.588640 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:58.588596    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret\") pod \"global-pull-secret-syncer-kj85s\" (UID: \"37ca8378-c8ef-4342-b489-631cba71f8c5\") " pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:17:58.589080 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:58.588693    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d84gb\" (UniqueName: \"kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb\") pod \"network-check-target-55qfp\" (UID: \"ac1f1f3d-3021-4a29-9162-77a512ca3a29\") " pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:17:58.589080 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:58.588774    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:58.589080 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:58.588851    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:58.589080 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:58.588865    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret podName:37ca8378-c8ef-4342-b489-631cba71f8c5 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:06.588844314 +0000 UTC m=+18.068238466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret") pod "global-pull-secret-syncer-kj85s" (UID: "37ca8378-c8ef-4342-b489-631cba71f8c5") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:58.589080 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:58.588872    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:58.589080 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:58.588886    2566 projected.go:196] Error preparing data for projected volume kube-api-access-d84gb for pod openshift-network-diagnostics/network-check-target-55qfp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:58.589080 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:58.588937    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb podName:ac1f1f3d-3021-4a29-9162-77a512ca3a29 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:06.588921564 +0000 UTC m=+18.068315705 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d84gb" (UniqueName: "kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb") pod "network-check-target-55qfp" (UID: "ac1f1f3d-3021-4a29-9162-77a512ca3a29") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:58.689920 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:58.689885    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs\") pod \"network-metrics-daemon-262h2\" (UID: \"e701e309-b8b1-47c5-927a-ce15959fafa1\") " pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:17:58.690089 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:58.690007    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:58.690089 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:58.690086    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs podName:e701e309-b8b1-47c5-927a-ce15959fafa1 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:06.690066316 +0000 UTC m=+18.169460463 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs") pod "network-metrics-daemon-262h2" (UID: "e701e309-b8b1-47c5-927a-ce15959fafa1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:59.043630 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:17:59.043333    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:17:59.043630 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:17:59.043455    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55qfp" podUID="ac1f1f3d-3021-4a29-9162-77a512ca3a29"
Sep 05 02:18:00.042628 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:00.042371    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:18:00.043087 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:00.042753    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-262h2" podUID="e701e309-b8b1-47c5-927a-ce15959fafa1"
Sep 05 02:18:00.043167 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:00.042574    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:18:00.043221 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:00.043204    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj85s" podUID="37ca8378-c8ef-4342-b489-631cba71f8c5"
Sep 05 02:18:01.042676 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:01.042638    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:18:01.043102 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:01.042777    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55qfp" podUID="ac1f1f3d-3021-4a29-9162-77a512ca3a29"
Sep 05 02:18:02.043106 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:02.043068    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:18:02.043562 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:02.043123    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:18:02.043562 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:02.043200    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-262h2" podUID="e701e309-b8b1-47c5-927a-ce15959fafa1"
Sep 05 02:18:02.043562 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:02.043318    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj85s" podUID="37ca8378-c8ef-4342-b489-631cba71f8c5"
Sep 05 02:18:03.042579 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:03.042548    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:18:03.042764 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:03.042723    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55qfp" podUID="ac1f1f3d-3021-4a29-9162-77a512ca3a29"
Sep 05 02:18:04.043189 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:04.043159    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:18:04.043605 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:04.043170    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:18:04.043605 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:04.043270    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj85s" podUID="37ca8378-c8ef-4342-b489-631cba71f8c5"
Sep 05 02:18:04.043605 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:04.043390    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-262h2" podUID="e701e309-b8b1-47c5-927a-ce15959fafa1"
Sep 05 02:18:05.043146 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:05.043112    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:18:05.043309 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:05.043237    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55qfp" podUID="ac1f1f3d-3021-4a29-9162-77a512ca3a29"
Sep 05 02:18:06.043082 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:06.043043    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:18:06.043289 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:06.043159    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj85s" podUID="37ca8378-c8ef-4342-b489-631cba71f8c5"
Sep 05 02:18:06.043289 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:06.043216    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:18:06.043666 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:06.043323    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-262h2" podUID="e701e309-b8b1-47c5-927a-ce15959fafa1"
Sep 05 02:18:06.649310 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:06.649278    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret\") pod \"global-pull-secret-syncer-kj85s\" (UID: \"37ca8378-c8ef-4342-b489-631cba71f8c5\") " pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:18:06.649483 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:06.649329    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d84gb\" (UniqueName: \"kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb\") pod \"network-check-target-55qfp\" (UID: \"ac1f1f3d-3021-4a29-9162-77a512ca3a29\") " pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:18:06.649483 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:06.649438    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:06.649483 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:06.649451    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:06.649483 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:06.649467    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:06.649483 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:06.649478    2566 projected.go:196] Error preparing data for projected volume kube-api-access-d84gb for pod openshift-network-diagnostics/network-check-target-55qfp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:06.649687 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:06.649508    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret podName:37ca8378-c8ef-4342-b489-631cba71f8c5 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:22.649490014 +0000 UTC m=+34.128884164 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret") pod "global-pull-secret-syncer-kj85s" (UID: "37ca8378-c8ef-4342-b489-631cba71f8c5") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:06.649687 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:06.649523    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb podName:ac1f1f3d-3021-4a29-9162-77a512ca3a29 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:22.649517387 +0000 UTC m=+34.128911523 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d84gb" (UniqueName: "kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb") pod "network-check-target-55qfp" (UID: "ac1f1f3d-3021-4a29-9162-77a512ca3a29") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:06.750501 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:06.750457    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs\") pod \"network-metrics-daemon-262h2\" (UID: \"e701e309-b8b1-47c5-927a-ce15959fafa1\") " pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:18:06.750690 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:06.750611    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:06.750690 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:06.750683    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs podName:e701e309-b8b1-47c5-927a-ce15959fafa1 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:22.750666377 +0000 UTC m=+34.230060528 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs") pod "network-metrics-daemon-262h2" (UID: "e701e309-b8b1-47c5-927a-ce15959fafa1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:07.042771 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:07.042683    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:18:07.042947 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:07.042806    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55qfp" podUID="ac1f1f3d-3021-4a29-9162-77a512ca3a29"
Sep 05 02:18:08.042762 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:08.042717    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:18:08.043216 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:08.042736    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:18:08.043216 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:08.042875    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-262h2" podUID="e701e309-b8b1-47c5-927a-ce15959fafa1"
Sep 05 02:18:08.043216 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:08.042944    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj85s" podUID="37ca8378-c8ef-4342-b489-631cba71f8c5"
Sep 05 02:18:09.043317 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:09.043289    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:18:09.043729 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:09.043392    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55qfp" podUID="ac1f1f3d-3021-4a29-9162-77a512ca3a29"
Sep 05 02:18:10.042683 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:10.042599    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:18:10.042755 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:10.042723    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj85s" podUID="37ca8378-c8ef-4342-b489-631cba71f8c5"
Sep 05 02:18:10.043174 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:10.043149    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:18:10.043290 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:10.043269    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-262h2" podUID="e701e309-b8b1-47c5-927a-ce15959fafa1"
Sep 05 02:18:10.182149 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:10.181825    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-shqpm" event={"ID":"eebe6f33-b592-4ee7-81e6-f3256af74fd4","Type":"ContainerStarted","Data":"223a8b0565776f0619ba5df2787ed4b40378f380a8f1205832b2e2fea9543f3e"}
Sep 05 02:18:10.186322 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:10.185948    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wxxsg" event={"ID":"974e8bfc-b802-4dfa-94c6-75a4331be5a5","Type":"ContainerStarted","Data":"6a0003ed142c6c0add89c77b532d44604f091817163dc88907b582e5f6de4040"}
Sep 05 02:18:10.190444 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:10.190367    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7" event={"ID":"f534411f-681d-4622-8b46-747603a90cdc","Type":"ContainerStarted","Data":"bd307c006fb7b682bb7b8fe8243930e7fbaa4925012cd5d8173fb7111f2a330f"}
Sep 05 02:18:10.192276 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:10.192223    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b89gp" event={"ID":"24ff133a-c6d2-4cbc-8dd8-a1e662c05baf","Type":"ContainerStarted","Data":"5f0b4045403ece1f10cafd408c42cd25a75b17a2d71b796ce56b9f3069f2c198"}
Sep 05 02:18:10.194861 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:10.194626    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf" event={"ID":"f16c6647-7313-4ab5-b757-d95b95979d3c","Type":"ContainerStarted","Data":"ccdd81d05ce31e36974f1b5af2b93404e4813e7015c33c81ca4ce046f49d537c"}
Sep 05 02:18:10.194861 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:10.194709    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf" event={"ID":"f16c6647-7313-4ab5-b757-d95b95979d3c","Type":"ContainerStarted","Data":"f34cc9a17f55e5bf8394832ee00252c47ba1afbf02c0d582e7c4bb264412144e"}
Sep 05 02:18:10.197020 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:10.196944    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-279ft" event={"ID":"4b864c4d-5153-4fed-a470-e6848614f613","Type":"ContainerStarted","Data":"d9e97cffb37996bb8af5d497063fd41dbac9d76a7e4c4698ccf1c6d74a8c5d85"}
Sep 05 02:18:10.202128 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:10.202102    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6wbh8" event={"ID":"5fa3cda0-ec69-4fe0-b459-119a5d7d4d42","Type":"ContainerStarted","Data":"a9a1decdf7646886c32c381bae360a39492793aab6a0ea66464f228f24a0b663"}
Sep 05 02:18:10.206879 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:10.206680    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vxccn" event={"ID":"c290fb8c-2fc3-4aff-863a-d8bf1ee41e01","Type":"ContainerStarted","Data":"473f8574702b9c362d638f1c06e934ba19639ee1f3c8142598b8e8dd31d59049"}
Sep 05 02:18:10.218654 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:10.218607    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-shqpm" podStartSLOduration=3.316464795 podStartE2EDuration="21.218590299s" totalImagesPullingTime="17.902125504s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:10.204092432 +0000 UTC m=+21.683486591" watchObservedRunningTime="2026-09-05 02:18:10.218590299 +0000 UTC m=+21.697984459"
Sep 05 02:18:10.218989 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:10.218958    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wxxsg" podStartSLOduration=3.589538217 podStartE2EDuration="21.218949821s" totalImagesPullingTime="17.629411604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:10.218784171 +0000 UTC m=+21.698178328" watchObservedRunningTime="2026-09-05 02:18:10.218949821 +0000 UTC m=+21.698343980"
Sep 05 02:18:10.233534 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:10.233494    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-b89gp" podStartSLOduration=3.349998051 podStartE2EDuration="21.233477931s" totalImagesPullingTime="17.88347988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:10.233465407 +0000 UTC m=+21.712859566" watchObservedRunningTime="2026-09-05 02:18:10.233477931 +0000 UTC m=+21.712872090"
Sep 05 02:18:10.250165 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:10.250126    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6wbh8" podStartSLOduration=3.277051617 podStartE2EDuration="21.25011238s" totalImagesPullingTime="17.973060763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:10.249898537 +0000 UTC m=+21.729292696" watchObservedRunningTime="2026-09-05 02:18:10.25011238 +0000 UTC m=+21.729506537"
Sep 05 02:18:10.284452 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:10.284412    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vxccn" podStartSLOduration=3.381306648 podStartE2EDuration="21.284397636s" totalImagesPullingTime="17.903090988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:10.28396136 +0000 UTC m=+21.763355519" watchObservedRunningTime="2026-09-05 02:18:10.284397636 +0000 UTC m=+21.763791844"
Sep 05 02:18:10.570887 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:10.570857    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wxxsg"
Sep 05 02:18:10.571342 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:10.571322    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wxxsg"
Sep 05 02:18:11.042395 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:11.042361    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:18:11.042635 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:11.042497    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55qfp" podUID="ac1f1f3d-3021-4a29-9162-77a512ca3a29"
Sep 05 02:18:11.212749 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:11.212713    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf" event={"ID":"f16c6647-7313-4ab5-b757-d95b95979d3c","Type":"ContainerStarted","Data":"df0fb35e34ebfae559797012ec0ed9410f805fb90ecec37f664e1a0f93829190"}
Sep 05 02:18:11.213581 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:11.212788    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf" event={"ID":"f16c6647-7313-4ab5-b757-d95b95979d3c","Type":"ContainerStarted","Data":"0a30e2f36b40975ee7b02c3395d4ec18508177f234c83224fc28f2682299a2b3"}
Sep 05 02:18:11.213581 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:11.212820    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf" event={"ID":"f16c6647-7313-4ab5-b757-d95b95979d3c","Type":"ContainerStarted","Data":"fa9433eea1652a8287e64ca46064828ed4a3539e3376ffa60b34003f09b7033f"}
Sep 05 02:18:11.213581 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:11.212871    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf" event={"ID":"f16c6647-7313-4ab5-b757-d95b95979d3c","Type":"ContainerStarted","Data":"3c40525e41cd55382ff8a9dd85df08f593a6e04d27b5b9dcc2563269bfdfc8da"}
Sep 05 02:18:11.215673 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:11.214939    2566 generic.go:356] "Generic (PLEG): container finished" podID="4b864c4d-5153-4fed-a470-e6848614f613" containerID="d9e97cffb37996bb8af5d497063fd41dbac9d76a7e4c4698ccf1c6d74a8c5d85" exitCode=0
Sep 05 02:18:11.215928 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:11.215902    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-279ft" event={"ID":"4b864c4d-5153-4fed-a470-e6848614f613","Type":"ContainerDied","Data":"d9e97cffb37996bb8af5d497063fd41dbac9d76a7e4c4698ccf1c6d74a8c5d85"}
Sep 05 02:18:11.216039 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:11.215957    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wxxsg"
Sep 05 02:18:11.216450 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:11.216425    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wxxsg"
Sep 05 02:18:11.692736 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:11.692592    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 05 02:18:11.960754 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:11.960648    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-05T02:18:11.692732322Z","UUID":"993e601a-9df6-420f-aec3-644941e6695e","Handler":null,"Name":"","Endpoint":""}
Sep 05 02:18:11.962272 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:11.962252    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 05 02:18:11.962423 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:11.962283    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 05 02:18:12.042422 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:12.042394    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:18:12.042591 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:12.042397    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:18:12.042591 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:12.042534    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-262h2" podUID="e701e309-b8b1-47c5-927a-ce15959fafa1"
Sep 05 02:18:12.042695 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:12.042604    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj85s" podUID="37ca8378-c8ef-4342-b489-631cba71f8c5"
Sep 05 02:18:12.220248 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:12.220161    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pp29d" event={"ID":"7e81e0ea-1618-456d-b0f4-33c3e3f3e518","Type":"ContainerStarted","Data":"72be1e6efd961fb8903c2665d532fc73c404d017e64cb4e624d115479b38e264"}
Sep 05 02:18:12.223418 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:12.223397    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7" event={"ID":"f534411f-681d-4622-8b46-747603a90cdc","Type":"ContainerStarted","Data":"10411b2f74159069d3bb51cf108e8c8c6b1777eaa6cff5277d1856a9998a6b71"}
Sep 05 02:18:12.236961 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:12.236919    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pp29d" podStartSLOduration=5.310419701 podStartE2EDuration="23.236905134s" totalImagesPullingTime="17.926485433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:12.236305324 +0000 UTC m=+23.715699485" watchObservedRunningTime="2026-09-05 02:18:12.236905134 +0000 UTC m=+23.716299292"
Sep 05 02:18:13.042547 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:13.042501    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:18:13.042777 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:13.042638    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55qfp" podUID="ac1f1f3d-3021-4a29-9162-77a512ca3a29"
Sep 05 02:18:13.227900 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:13.227865    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7" event={"ID":"f534411f-681d-4622-8b46-747603a90cdc","Type":"ContainerStarted","Data":"16efe0c5d2671b49e4f55a878783b80103537b887f59b6bb3ee3511313eff190"}
Sep 05 02:18:13.231728 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:13.231702    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf" event={"ID":"f16c6647-7313-4ab5-b757-d95b95979d3c","Type":"ContainerStarted","Data":"88e53d2541b0f6d1f3057a6bd20d7254f813b92a8e4e08ea618bb47011483f54"}
Sep 05 02:18:13.246646 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:13.246602    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s96x7" podStartSLOduration=4.123442114 podStartE2EDuration="24.246583818s" totalImagesPullingTime="20.123141704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:13.24508759 +0000 UTC m=+24.724481751" watchObservedRunningTime="2026-09-05 02:18:13.246583818 +0000 UTC m=+24.725977976"
Sep 05 02:18:13.574822 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:13.574785    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:13.660123 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:13.660094    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:14.042367 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:14.042332    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:18:14.042563 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:14.042450    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj85s" podUID="37ca8378-c8ef-4342-b489-631cba71f8c5"
Sep 05 02:18:14.042563 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:14.042503    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:18:14.042644 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:14.042620    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-262h2" podUID="e701e309-b8b1-47c5-927a-ce15959fafa1"
Sep 05 02:18:15.043312 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:15.043224    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:18:15.043702 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:15.043365    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55qfp" podUID="ac1f1f3d-3021-4a29-9162-77a512ca3a29"
Sep 05 02:18:16.042598 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:16.042412    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:18:16.042773 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:16.042419    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:18:16.042773 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:16.042668    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj85s" podUID="37ca8378-c8ef-4342-b489-631cba71f8c5"
Sep 05 02:18:16.042773 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:16.042751    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-262h2" podUID="e701e309-b8b1-47c5-927a-ce15959fafa1"
Sep 05 02:18:16.241297 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:16.241259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf" event={"ID":"f16c6647-7313-4ab5-b757-d95b95979d3c","Type":"ContainerStarted","Data":"5f0c928a56ff1d21830d508a49fecc3a23c675a1e690415ae8023676ed7dab12"}
Sep 05 02:18:16.241992 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:16.241514    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:18:16.241992 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:16.241592    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:18:16.243473 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:16.243448    2566 generic.go:356] "Generic (PLEG): container finished" podID="4b864c4d-5153-4fed-a470-e6848614f613" containerID="8a058e09d6f13d76087cf1279e9f1ab8f8136ffcf61bbd61640705f041d66a20" exitCode=0
Sep 05 02:18:16.243611 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:16.243495    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-279ft" event={"ID":"4b864c4d-5153-4fed-a470-e6848614f613","Type":"ContainerDied","Data":"8a058e09d6f13d76087cf1279e9f1ab8f8136ffcf61bbd61640705f041d66a20"}
Sep 05 02:18:16.256811 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:16.256789    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:18:16.256985 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:16.256968    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:18:16.269106 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:16.269062    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf" podStartSLOduration=9.314019998 podStartE2EDuration="27.269049406s" totalImagesPullingTime="17.955029408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:16.268935117 +0000 UTC m=+27.748329276" watchObservedRunningTime="2026-09-05 02:18:16.269049406 +0000 UTC m=+27.748443564"
Sep 05 02:18:17.030260 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:17.030235    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:17.043245 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:17.043219    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:18:17.043394 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:17.043321    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55qfp" podUID="ac1f1f3d-3021-4a29-9162-77a512ca3a29"
Sep 05 02:18:17.130713 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:17.130498    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:17.247250 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:17.247217    2566 generic.go:356] "Generic (PLEG): container finished" podID="4b864c4d-5153-4fed-a470-e6848614f613" containerID="73292e870196c4330b6c51452b47da0a1ed4a5184563876a867e952c4db86f34" exitCode=0
Sep 05 02:18:17.247714 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:17.247279    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-279ft" event={"ID":"4b864c4d-5153-4fed-a470-e6848614f613","Type":"ContainerDied","Data":"73292e870196c4330b6c51452b47da0a1ed4a5184563876a867e952c4db86f34"}
Sep 05 02:18:17.247714 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:17.247368    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 05 02:18:17.269091 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:17.268414    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-55qfp"]
Sep 05 02:18:17.269091 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:17.269047    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:18:17.269252 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:17.269190    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55qfp" podUID="ac1f1f3d-3021-4a29-9162-77a512ca3a29"
Sep 05 02:18:17.270960 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:17.270713    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kj85s"]
Sep 05 02:18:17.270960 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:17.270810    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:18:17.270960 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:17.270922    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj85s" podUID="37ca8378-c8ef-4342-b489-631cba71f8c5"
Sep 05 02:18:17.271488 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:17.271463    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-262h2"]
Sep 05 02:18:17.271589 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:17.271566    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:18:17.271676 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:17.271654    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-262h2" podUID="e701e309-b8b1-47c5-927a-ce15959fafa1"
Sep 05 02:18:17.445641 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:17.445606    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:17.513320 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:17.513294    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:18.252030 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:18.251993    2566 generic.go:356] "Generic (PLEG): container finished" podID="4b864c4d-5153-4fed-a470-e6848614f613" containerID="48c20518e9b2cc49ce09478a856b4cd2e7f65456de4cd1e5dd44ddd7f3a741aa" exitCode=0
Sep 05 02:18:18.252386 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:18.252068    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-279ft" event={"ID":"4b864c4d-5153-4fed-a470-e6848614f613","Type":"ContainerDied","Data":"48c20518e9b2cc49ce09478a856b4cd2e7f65456de4cd1e5dd44ddd7f3a741aa"}
Sep 05 02:18:18.252386 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:18.252159    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 05 02:18:18.549652 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:18.549575    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:18:18.748227 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:18.748188    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:19.043895 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:19.043856    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:18:19.043895 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:19.043890    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:18:19.044113 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:19.043968    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:18:19.044113 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:19.044006    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55qfp" podUID="ac1f1f3d-3021-4a29-9162-77a512ca3a29"
Sep 05 02:18:19.044113 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:19.044069    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj85s" podUID="37ca8378-c8ef-4342-b489-631cba71f8c5"
Sep 05 02:18:19.044242 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:19.044166    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-262h2" podUID="e701e309-b8b1-47c5-927a-ce15959fafa1"
Sep 05 02:18:19.387734 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:19.387700    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:19.453913 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:19.453871    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:20.378654 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:20.378619    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:20.458788 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:20.458754    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:21.043072 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:21.043045    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:18:21.043200 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:21.043173    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-55qfp" podUID="ac1f1f3d-3021-4a29-9162-77a512ca3a29"
Sep 05 02:18:21.043282 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:21.043260    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:18:21.043338 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:21.043297    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:18:21.043454 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:21.043413    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-262h2" podUID="e701e309-b8b1-47c5-927a-ce15959fafa1"
Sep 05 02:18:21.043520 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:21.043487    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj85s" podUID="37ca8378-c8ef-4342-b489-631cba71f8c5"
Sep 05 02:18:22.669255 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.669202    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d84gb\" (UniqueName: \"kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb\") pod \"network-check-target-55qfp\" (UID: \"ac1f1f3d-3021-4a29-9162-77a512ca3a29\") " pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:18:22.669255 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.669244    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret\") pod \"global-pull-secret-syncer-kj85s\" (UID: \"37ca8378-c8ef-4342-b489-631cba71f8c5\") " pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:18:22.669643 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:22.669342    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:22.669643 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:22.669368    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:22.669643 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:22.669387    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:22.669643 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:22.669391    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret podName:37ca8378-c8ef-4342-b489-631cba71f8c5 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:54.66937868 +0000 UTC m=+66.148772820 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret") pod "global-pull-secret-syncer-kj85s" (UID: "37ca8378-c8ef-4342-b489-631cba71f8c5") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:22.669643 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:22.669397    2566 projected.go:196] Error preparing data for projected volume kube-api-access-d84gb for pod openshift-network-diagnostics/network-check-target-55qfp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:22.669643 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:22.669441    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb podName:ac1f1f3d-3021-4a29-9162-77a512ca3a29 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:54.669428292 +0000 UTC m=+66.148822428 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-d84gb" (UniqueName: "kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb") pod "network-check-target-55qfp" (UID: "ac1f1f3d-3021-4a29-9162-77a512ca3a29") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:22.734232 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.734206    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeReady"
Sep 05 02:18:22.734392 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.734313    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 05 02:18:22.769990 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.769961    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs\") pod \"network-metrics-daemon-262h2\" (UID: \"e701e309-b8b1-47c5-927a-ce15959fafa1\") " pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:18:22.770126 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:22.770101    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:22.770165 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:22.770155    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs podName:e701e309-b8b1-47c5-927a-ce15959fafa1 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:54.770141975 +0000 UTC m=+66.249536111 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs") pod "network-metrics-daemon-262h2" (UID: "e701e309-b8b1-47c5-927a-ce15959fafa1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:22.781362 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.781337    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-66mtm"]
Sep 05 02:18:22.792999 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.792979    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zgfdl"]
Sep 05 02:18:22.793107 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.793090    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-66mtm"
Sep 05 02:18:22.795840 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.795807    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 05 02:18:22.795918 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.795808    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 05 02:18:22.800619 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.799682    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cl7gt\""
Sep 05 02:18:22.805221 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.805203    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-66mtm"]
Sep 05 02:18:22.805305 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.805233    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zgfdl"]
Sep 05 02:18:22.805344 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.805305    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zgfdl"
Sep 05 02:18:22.811110 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.811087    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 05 02:18:22.811193 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.811087    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cd6qr\""
Sep 05 02:18:22.811193 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.811182    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 05 02:18:22.811403 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.811390    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 05 02:18:22.871342 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.871183    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqm48\" (UniqueName: \"kubernetes.io/projected/18051063-ebe9-4362-ab51-4a4da32e15ff-kube-api-access-lqm48\") pod \"dns-default-66mtm\" (UID: \"18051063-ebe9-4362-ab51-4a4da32e15ff\") " pod="openshift-dns/dns-default-66mtm"
Sep 05 02:18:22.871494 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.871370    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/18051063-ebe9-4362-ab51-4a4da32e15ff-config-volume\") pod \"dns-default-66mtm\" (UID: \"18051063-ebe9-4362-ab51-4a4da32e15ff\") " pod="openshift-dns/dns-default-66mtm"
Sep 05 02:18:22.871494 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.871404    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls\") pod \"dns-default-66mtm\" (UID: \"18051063-ebe9-4362-ab51-4a4da32e15ff\") " pod="openshift-dns/dns-default-66mtm"
Sep 05 02:18:22.871494 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.871431    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18051063-ebe9-4362-ab51-4a4da32e15ff-tmp-dir\") pod \"dns-default-66mtm\" (UID: \"18051063-ebe9-4362-ab51-4a4da32e15ff\") " pod="openshift-dns/dns-default-66mtm"
Sep 05 02:18:22.971917 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.971810    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls\") pod \"dns-default-66mtm\" (UID: \"18051063-ebe9-4362-ab51-4a4da32e15ff\") " pod="openshift-dns/dns-default-66mtm"
Sep 05 02:18:22.971917 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.971888    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18051063-ebe9-4362-ab51-4a4da32e15ff-tmp-dir\") pod \"dns-default-66mtm\" (UID: \"18051063-ebe9-4362-ab51-4a4da32e15ff\") " pod="openshift-dns/dns-default-66mtm"
Sep 05 02:18:22.971917 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.971910    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b81f383-6a65-49a2-a8d8-c5efe33ffe3d-cert\") pod \"ingress-canary-zgfdl\" (UID: \"6b81f383-6a65-49a2-a8d8-c5efe33ffe3d\") " pod="openshift-ingress-canary/ingress-canary-zgfdl"
Sep 05 02:18:22.972269 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.971932    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqm48\" (UniqueName: \"kubernetes.io/projected/18051063-ebe9-4362-ab51-4a4da32e15ff-kube-api-access-lqm48\") pod \"dns-default-66mtm\" (UID: \"18051063-ebe9-4362-ab51-4a4da32e15ff\") " pod="openshift-dns/dns-default-66mtm"
Sep 05 02:18:22.972269 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:22.971968    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:22.972269 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.972016    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/18051063-ebe9-4362-ab51-4a4da32e15ff-config-volume\") pod \"dns-default-66mtm\" (UID: \"18051063-ebe9-4362-ab51-4a4da32e15ff\") " pod="openshift-dns/dns-default-66mtm"
Sep 05 02:18:22.972269 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:22.972042    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls podName:18051063-ebe9-4362-ab51-4a4da32e15ff nodeName:}" failed. No retries permitted until 2026-09-05 02:18:23.472020064 +0000 UTC m=+34.951414214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls") pod "dns-default-66mtm" (UID: "18051063-ebe9-4362-ab51-4a4da32e15ff") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:22.972269 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.972097    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzktd\" (UniqueName: \"kubernetes.io/projected/6b81f383-6a65-49a2-a8d8-c5efe33ffe3d-kube-api-access-nzktd\") pod \"ingress-canary-zgfdl\" (UID: \"6b81f383-6a65-49a2-a8d8-c5efe33ffe3d\") " pod="openshift-ingress-canary/ingress-canary-zgfdl"
Sep 05 02:18:22.972457 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.972275    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18051063-ebe9-4362-ab51-4a4da32e15ff-tmp-dir\") pod \"dns-default-66mtm\" (UID: \"18051063-ebe9-4362-ab51-4a4da32e15ff\") " pod="openshift-dns/dns-default-66mtm"
Sep 05 02:18:22.972457 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.972428    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/18051063-ebe9-4362-ab51-4a4da32e15ff-config-volume\") pod \"dns-default-66mtm\" (UID: \"18051063-ebe9-4362-ab51-4a4da32e15ff\") " pod="openshift-dns/dns-default-66mtm"
Sep 05 02:18:22.984601 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.984575    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqm48\" (UniqueName: \"kubernetes.io/projected/18051063-ebe9-4362-ab51-4a4da32e15ff-kube-api-access-lqm48\") pod \"dns-default-66mtm\" (UID: \"18051063-ebe9-4362-ab51-4a4da32e15ff\") " pod="openshift-dns/dns-default-66mtm"
Sep 05 02:18:22.989572 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:22.989547    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:23.042741 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:23.042708    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:18:23.042961 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:23.042708    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:18:23.042961 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:23.042708    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:18:23.045007 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:23.044988    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4nzpg\""
Sep 05 02:18:23.045296 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:23.045276    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 05 02:18:23.045296 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:23.045287    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 05 02:18:23.045636 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:23.045616    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hk4wx\""
Sep 05 02:18:23.045636 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:23.045643    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 05 02:18:23.045792 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:23.045679    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 05 02:18:23.072364 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:23.072343    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b81f383-6a65-49a2-a8d8-c5efe33ffe3d-cert\") pod \"ingress-canary-zgfdl\" (UID: \"6b81f383-6a65-49a2-a8d8-c5efe33ffe3d\") " pod="openshift-ingress-canary/ingress-canary-zgfdl"
Sep 05 02:18:23.072475 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:23.072400    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzktd\" (UniqueName: \"kubernetes.io/projected/6b81f383-6a65-49a2-a8d8-c5efe33ffe3d-kube-api-access-nzktd\") pod \"ingress-canary-zgfdl\" (UID: \"6b81f383-6a65-49a2-a8d8-c5efe33ffe3d\") " pod="openshift-ingress-canary/ingress-canary-zgfdl"
Sep 05 02:18:23.074785 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:23.074760    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b81f383-6a65-49a2-a8d8-c5efe33ffe3d-cert\") pod \"ingress-canary-zgfdl\" (UID: \"6b81f383-6a65-49a2-a8d8-c5efe33ffe3d\") " pod="openshift-ingress-canary/ingress-canary-zgfdl"
Sep 05 02:18:23.080313 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:23.080290    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzktd\" (UniqueName: \"kubernetes.io/projected/6b81f383-6a65-49a2-a8d8-c5efe33ffe3d-kube-api-access-nzktd\") pod \"ingress-canary-zgfdl\" (UID: \"6b81f383-6a65-49a2-a8d8-c5efe33ffe3d\") " pod="openshift-ingress-canary/ingress-canary-zgfdl"
Sep 05 02:18:23.089631 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:23.089610    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:23.112763 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:23.112742    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zgfdl"
Sep 05 02:18:23.268029 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:23.267947    2566 generic.go:356] "Generic (PLEG): container finished" podID="4b864c4d-5153-4fed-a470-e6848614f613" containerID="994de526b6275f2ab7d3a8bb28c89fa6f4e65a846abddceb756bfdcbc128904d" exitCode=0
Sep 05 02:18:23.268029 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:23.268000    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-279ft" event={"ID":"4b864c4d-5153-4fed-a470-e6848614f613","Type":"ContainerDied","Data":"994de526b6275f2ab7d3a8bb28c89fa6f4e65a846abddceb756bfdcbc128904d"}
Sep 05 02:18:23.411851 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:23.411802    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zgfdl"]
Sep 05 02:18:23.416877 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:18:23.416848    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b81f383_6a65_49a2_a8d8_c5efe33ffe3d.slice/crio-39d53462b10a55fa8864002794f4992072e46d2e2d01086f9f6a0a70338ccbcf WatchSource:0}: Error finding container 39d53462b10a55fa8864002794f4992072e46d2e2d01086f9f6a0a70338ccbcf: Status 404 returned error can't find the container with id 39d53462b10a55fa8864002794f4992072e46d2e2d01086f9f6a0a70338ccbcf
Sep 05 02:18:23.476623 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:23.476593    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls\") pod \"dns-default-66mtm\" (UID: \"18051063-ebe9-4362-ab51-4a4da32e15ff\") " pod="openshift-dns/dns-default-66mtm"
Sep 05 02:18:23.476770 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:23.476713    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:23.476770 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:23.476763    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls podName:18051063-ebe9-4362-ab51-4a4da32e15ff nodeName:}" failed. No retries permitted until 2026-09-05 02:18:24.476749698 +0000 UTC m=+35.956143835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls") pod "dns-default-66mtm" (UID: "18051063-ebe9-4362-ab51-4a4da32e15ff") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:24.272068 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:24.272031    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zgfdl" event={"ID":"6b81f383-6a65-49a2-a8d8-c5efe33ffe3d","Type":"ContainerStarted","Data":"39d53462b10a55fa8864002794f4992072e46d2e2d01086f9f6a0a70338ccbcf"}
Sep 05 02:18:24.275688 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:24.275657    2566 generic.go:356] "Generic (PLEG): container finished" podID="4b864c4d-5153-4fed-a470-e6848614f613" containerID="5f674b1516ac0eee6f5125b1b606a0e4b0837941bb63c35d98db878ebe0b18d9" exitCode=0
Sep 05 02:18:24.275814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:24.275708    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-279ft" event={"ID":"4b864c4d-5153-4fed-a470-e6848614f613","Type":"ContainerDied","Data":"5f674b1516ac0eee6f5125b1b606a0e4b0837941bb63c35d98db878ebe0b18d9"}
Sep 05 02:18:24.393539 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:24.393504    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:24.482770 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:24.482731    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls\") pod \"dns-default-66mtm\" (UID: \"18051063-ebe9-4362-ab51-4a4da32e15ff\") " pod="openshift-dns/dns-default-66mtm"
Sep 05 02:18:24.482935 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:24.482895    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:24.483001 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:24.482962    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls podName:18051063-ebe9-4362-ab51-4a4da32e15ff nodeName:}" failed. No retries permitted until 2026-09-05 02:18:26.482940841 +0000 UTC m=+37.962334991 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls") pod "dns-default-66mtm" (UID: "18051063-ebe9-4362-ab51-4a4da32e15ff") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:25.282134 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:25.282095    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-279ft" event={"ID":"4b864c4d-5153-4fed-a470-e6848614f613","Type":"ContainerStarted","Data":"a7f21c6d41728dac482aabfb6f166cd87c0b5c5b0b547d9b017b9aa518774230"}
Sep 05 02:18:25.305980 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:25.305926    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-279ft" podStartSLOduration=8.698689853 podStartE2EDuration="36.305910245s" totalImagesPullingTime="27.607220392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:25.30486811 +0000 UTC m=+36.784262269" watchObservedRunningTime="2026-09-05 02:18:25.305910245 +0000 UTC m=+36.785304404"
Sep 05 02:18:25.428457 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:25.428423    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:25.492199 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:25.492160    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:26.286218 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:26.286134    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zgfdl" event={"ID":"6b81f383-6a65-49a2-a8d8-c5efe33ffe3d","Type":"ContainerStarted","Data":"a020d1e073a85d8bef4aac74b4c540c124e9b4b9e7ae749f79e32379f31812b1"}
Sep 05 02:18:26.301036 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:26.300989    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zgfdl" podStartSLOduration=1.716884559 podStartE2EDuration="4.300970727s" totalImagesPullingTime="2.584086168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:26.300549203 +0000 UTC m=+37.779943361" watchObservedRunningTime="2026-09-05 02:18:26.300970727 +0000 UTC m=+37.780364885"
Sep 05 02:18:26.499254 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:26.499217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls\") pod \"dns-default-66mtm\" (UID: \"18051063-ebe9-4362-ab51-4a4da32e15ff\") " pod="openshift-dns/dns-default-66mtm"
Sep 05 02:18:26.499468 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:26.499388    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:26.499530 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:26.499465    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls podName:18051063-ebe9-4362-ab51-4a4da32e15ff nodeName:}" failed. No retries permitted until 2026-09-05 02:18:30.499445769 +0000 UTC m=+41.978839905 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls") pod "dns-default-66mtm" (UID: "18051063-ebe9-4362-ab51-4a4da32e15ff") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:28.401943 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:28.401909    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:29.793132 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:29.793090    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:30.527173 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:30.527129    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls\") pod \"dns-default-66mtm\" (UID: \"18051063-ebe9-4362-ab51-4a4da32e15ff\") " pod="openshift-dns/dns-default-66mtm"
Sep 05 02:18:30.527345 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:30.527283    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:30.527387 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:30.527349    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls podName:18051063-ebe9-4362-ab51-4a4da32e15ff nodeName:}" failed. No retries permitted until 2026-09-05 02:18:38.527333159 +0000 UTC m=+50.006727295 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls") pod "dns-default-66mtm" (UID: "18051063-ebe9-4362-ab51-4a4da32e15ff") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:32.451109 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:32.451072    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:32.506376 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:32.506349    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:34.459958 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:34.459925    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:34.528777 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:34.528753    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:36.679601 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:36.679568    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:36.792661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:36.792633    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:38.580317 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:38.580279    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls\") pod \"dns-default-66mtm\" (UID: \"18051063-ebe9-4362-ab51-4a4da32e15ff\") " pod="openshift-dns/dns-default-66mtm"
Sep 05 02:18:38.580679 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:38.580456    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:38.580679 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:38.580530    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls podName:18051063-ebe9-4362-ab51-4a4da32e15ff nodeName:}" failed. No retries permitted until 2026-09-05 02:18:54.580514589 +0000 UTC m=+66.059908729 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls") pod "dns-default-66mtm" (UID: "18051063-ebe9-4362-ab51-4a4da32e15ff") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:39.437280 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:39.437244    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:39.501341 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:39.501301    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:49.264670 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:49.264640    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7vnhf"
Sep 05 02:18:49.415173 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:49.415127    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:49.472822 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:49.472791    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:52.748915 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.748876    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-4r72s"]
Sep 05 02:18:52.786264 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.786232    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-4r72s"]
Sep 05 02:18:52.786428 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.786342    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4r72s"
Sep 05 02:18:52.788808 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.788783    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-dwmgs\""
Sep 05 02:18:52.789582 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.789555    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 05 02:18:52.789649 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.789555    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 05 02:18:52.789649 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.789557    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 05 02:18:52.789649 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.789563    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:18:52.882659 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.882622    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b2438b69-c255-4a19-98d7-56ab067c6a22-serving-cert\") pod \"service-ca-operator-859f94c4b-4r72s\" (UID: \"b2438b69-c255-4a19-98d7-56ab067c6a22\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4r72s"
Sep 05 02:18:52.882822 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.882663    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2438b69-c255-4a19-98d7-56ab067c6a22-config\") pod \"service-ca-operator-859f94c4b-4r72s\" (UID: \"b2438b69-c255-4a19-98d7-56ab067c6a22\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4r72s"
Sep 05 02:18:52.882822 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.882692    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4pxr\" (UniqueName: \"kubernetes.io/projected/b2438b69-c255-4a19-98d7-56ab067c6a22-kube-api-access-n4pxr\") pod \"service-ca-operator-859f94c4b-4r72s\" (UID: \"b2438b69-c255-4a19-98d7-56ab067c6a22\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4r72s"
Sep 05 02:18:52.944879 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.944846    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b77fd6b6-n5m6j"]
Sep 05 02:18:52.961925 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.961897    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b77fd6b6-n5m6j"]
Sep 05 02:18:52.962051 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.961985    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:52.964290 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.964260    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 05 02:18:52.964422 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.964305    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 05 02:18:52.964422 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.964372    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 05 02:18:52.964814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.964796    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-f79x9\""
Sep 05 02:18:52.970205 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.970187    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 05 02:18:52.983489 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.983470    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-certificates\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:52.983568 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.983518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73de4666-9d9c-482a-9f2f-b56dff7dec5b-image-registry-private-configuration\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:52.983568 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.983556    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-tls\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:52.983637 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.983583    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b2438b69-c255-4a19-98d7-56ab067c6a22-serving-cert\") pod \"service-ca-operator-859f94c4b-4r72s\" (UID: \"b2438b69-c255-4a19-98d7-56ab067c6a22\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4r72s"
Sep 05 02:18:52.983637 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.983630    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2438b69-c255-4a19-98d7-56ab067c6a22-config\") pod \"service-ca-operator-859f94c4b-4r72s\" (UID: \"b2438b69-c255-4a19-98d7-56ab067c6a22\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4r72s"
Sep 05 02:18:52.983698 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.983653    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73de4666-9d9c-482a-9f2f-b56dff7dec5b-installation-pull-secrets\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:52.983698 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.983671    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-bound-sa-token\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:52.983698 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.983690    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4pxr\" (UniqueName: \"kubernetes.io/projected/b2438b69-c255-4a19-98d7-56ab067c6a22-kube-api-access-n4pxr\") pod \"service-ca-operator-859f94c4b-4r72s\" (UID: \"b2438b69-c255-4a19-98d7-56ab067c6a22\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4r72s"
Sep 05 02:18:52.983801 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.983719    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73de4666-9d9c-482a-9f2f-b56dff7dec5b-trusted-ca\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:52.983801 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.983768    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73de4666-9d9c-482a-9f2f-b56dff7dec5b-ca-trust-extracted\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:52.983801 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.983791    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhgf6\" (UniqueName: \"kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-kube-api-access-lhgf6\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:52.984146 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.984129    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2438b69-c255-4a19-98d7-56ab067c6a22-config\") pod \"service-ca-operator-859f94c4b-4r72s\" (UID: \"b2438b69-c255-4a19-98d7-56ab067c6a22\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4r72s"
Sep 05 02:18:52.986868 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.986848    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b2438b69-c255-4a19-98d7-56ab067c6a22-serving-cert\") pod \"service-ca-operator-859f94c4b-4r72s\" (UID: \"b2438b69-c255-4a19-98d7-56ab067c6a22\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4r72s"
Sep 05 02:18:52.998371 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:52.998348    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4pxr\" (UniqueName: \"kubernetes.io/projected/b2438b69-c255-4a19-98d7-56ab067c6a22-kube-api-access-n4pxr\") pod \"service-ca-operator-859f94c4b-4r72s\" (UID: \"b2438b69-c255-4a19-98d7-56ab067c6a22\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4r72s"
Sep 05 02:18:53.015517 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:53.015467    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:53.084815 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:53.084785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73de4666-9d9c-482a-9f2f-b56dff7dec5b-trusted-ca\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:53.084958 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:53.084853    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73de4666-9d9c-482a-9f2f-b56dff7dec5b-ca-trust-extracted\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:53.084958 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:53.084881    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhgf6\" (UniqueName: \"kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-kube-api-access-lhgf6\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:53.084958 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:53.084926    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-certificates\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:53.085108 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:53.085079    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73de4666-9d9c-482a-9f2f-b56dff7dec5b-image-registry-private-configuration\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:53.085155 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:53.085115    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-tls\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:53.085254 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:53.085239    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:53.085314 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:53.085258    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b77fd6b6-n5m6j: secret "image-registry-tls" not found
Sep 05 02:18:53.085314 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:53.085269    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73de4666-9d9c-482a-9f2f-b56dff7dec5b-ca-trust-extracted\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:53.085314 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:53.085240    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73de4666-9d9c-482a-9f2f-b56dff7dec5b-installation-pull-secrets\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:53.085314 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:53.085313    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-tls podName:73de4666-9d9c-482a-9f2f-b56dff7dec5b nodeName:}" failed. No retries permitted until 2026-09-05 02:18:53.585293701 +0000 UTC m=+65.064687839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-tls") pod "image-registry-b77fd6b6-n5m6j" (UID: "73de4666-9d9c-482a-9f2f-b56dff7dec5b") : secret "image-registry-tls" not found
Sep 05 02:18:53.085514 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:53.085341    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-bound-sa-token\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:53.085592 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:53.085574    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-certificates\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:53.085762 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:53.085742    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73de4666-9d9c-482a-9f2f-b56dff7dec5b-trusted-ca\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:53.087522 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:53.087497    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73de4666-9d9c-482a-9f2f-b56dff7dec5b-installation-pull-secrets\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:53.087597 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:53.087533    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73de4666-9d9c-482a-9f2f-b56dff7dec5b-image-registry-private-configuration\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:53.093093 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:53.093074    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-bound-sa-token\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:53.095144 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:53.095124    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4r72s"
Sep 05 02:18:53.095755 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:53.095737    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhgf6\" (UniqueName: \"kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-kube-api-access-lhgf6\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:53.115985 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:53.115953    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:53.386134 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:53.386105    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-4r72s"]
Sep 05 02:18:53.389103 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:18:53.389084    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2438b69_c255_4a19_98d7_56ab067c6a22.slice/crio-bbfaff5684a15efb17ed9682e5172c51b2193a79f80b38c48536ac6c90f51626 WatchSource:0}: Error finding container bbfaff5684a15efb17ed9682e5172c51b2193a79f80b38c48536ac6c90f51626: Status 404 returned error can't find the container with id bbfaff5684a15efb17ed9682e5172c51b2193a79f80b38c48536ac6c90f51626
Sep 05 02:18:53.588559 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:53.588526    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-tls\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:53.588725 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:53.588638    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:53.588725 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:53.588650    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b77fd6b6-n5m6j: secret "image-registry-tls" not found
Sep 05 02:18:53.588725 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:53.588696    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-tls podName:73de4666-9d9c-482a-9f2f-b56dff7dec5b nodeName:}" failed. No retries permitted until 2026-09-05 02:18:54.58868344 +0000 UTC m=+66.068077580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-tls") pod "image-registry-b77fd6b6-n5m6j" (UID: "73de4666-9d9c-482a-9f2f-b56dff7dec5b") : secret "image-registry-tls" not found
Sep 05 02:18:54.355375 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:54.355338    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4r72s" event={"ID":"b2438b69-c255-4a19-98d7-56ab067c6a22","Type":"ContainerStarted","Data":"bbfaff5684a15efb17ed9682e5172c51b2193a79f80b38c48536ac6c90f51626"}
Sep 05 02:18:54.596170 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:54.596131    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls\") pod \"dns-default-66mtm\" (UID: \"18051063-ebe9-4362-ab51-4a4da32e15ff\") " pod="openshift-dns/dns-default-66mtm"
Sep 05 02:18:54.596343 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:54.596182    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-tls\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:54.596343 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:54.596294    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:54.596343 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:54.596295    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:54.596474 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:54.596372    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls podName:18051063-ebe9-4362-ab51-4a4da32e15ff nodeName:}" failed. No retries permitted until 2026-09-05 02:19:26.596355465 +0000 UTC m=+98.075749605 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls") pod "dns-default-66mtm" (UID: "18051063-ebe9-4362-ab51-4a4da32e15ff") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:54.596474 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:54.596306    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b77fd6b6-n5m6j: secret "image-registry-tls" not found
Sep 05 02:18:54.596564 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:54.596483    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-tls podName:73de4666-9d9c-482a-9f2f-b56dff7dec5b nodeName:}" failed. No retries permitted until 2026-09-05 02:18:56.596467975 +0000 UTC m=+68.075862125 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-tls") pod "image-registry-b77fd6b6-n5m6j" (UID: "73de4666-9d9c-482a-9f2f-b56dff7dec5b") : secret "image-registry-tls" not found
Sep 05 02:18:54.697127 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:54.697092    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d84gb\" (UniqueName: \"kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb\") pod \"network-check-target-55qfp\" (UID: \"ac1f1f3d-3021-4a29-9162-77a512ca3a29\") " pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:18:54.697314 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:54.697175    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret\") pod \"global-pull-secret-syncer-kj85s\" (UID: \"37ca8378-c8ef-4342-b489-631cba71f8c5\") " pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:18:54.699636 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:54.699613    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 05 02:18:54.699636 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:54.699627    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 05 02:18:54.709541 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:54.709519    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 05 02:18:54.710025 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:54.710005    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ca8378-c8ef-4342-b489-631cba71f8c5-original-pull-secret\") pod \"global-pull-secret-syncer-kj85s\" (UID: \"37ca8378-c8ef-4342-b489-631cba71f8c5\") " pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:18:54.720818 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:54.720797    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d84gb\" (UniqueName: \"kubernetes.io/projected/ac1f1f3d-3021-4a29-9162-77a512ca3a29-kube-api-access-d84gb\") pod \"network-check-target-55qfp\" (UID: \"ac1f1f3d-3021-4a29-9162-77a512ca3a29\") " pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:18:54.797857 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:54.797816    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs\") pod \"network-metrics-daemon-262h2\" (UID: \"e701e309-b8b1-47c5-927a-ce15959fafa1\") " pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:18:54.800373 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:54.800348    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 05 02:18:54.808295 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:54.808274    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 05 02:18:54.808405 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:54.808333    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs podName:e701e309-b8b1-47c5-927a-ce15959fafa1 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:58.808315047 +0000 UTC m=+130.287709183 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs") pod "network-metrics-daemon-262h2" (UID: "e701e309-b8b1-47c5-927a-ce15959fafa1") : secret "metrics-daemon-secret" not found
Sep 05 02:18:54.858940 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:54.858898    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj85s"
Sep 05 02:18:54.865410 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:54.865384    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hk4wx\""
Sep 05 02:18:54.873756 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:54.873731    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:18:55.359963 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:55.359934    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4r72s" event={"ID":"b2438b69-c255-4a19-98d7-56ab067c6a22","Type":"ContainerStarted","Data":"b2103636498af8d6633988618f015652475025aebfb11b9f5e4f5d3d156d7c2b"}
Sep 05 02:18:55.377301 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:55.377248    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4r72s" podStartSLOduration=1.514986366 podStartE2EDuration="3.377232605s" totalImagesPullingTime="1.862246239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:55.376785681 +0000 UTC m=+66.856179852" watchObservedRunningTime="2026-09-05 02:18:55.377232605 +0000 UTC m=+66.856626744"
Sep 05 02:18:55.499374 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:55.499294    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-55qfp"]
Sep 05 02:18:55.500999 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:55.500977    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kj85s"]
Sep 05 02:18:55.503269 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:18:55.503240    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1f1f3d_3021_4a29_9162_77a512ca3a29.slice/crio-f78b0f392e2f6e00a1d27813d94d898939c4aad49917edbd6b1540223620a798 WatchSource:0}: Error finding container f78b0f392e2f6e00a1d27813d94d898939c4aad49917edbd6b1540223620a798: Status 404 returned error can't find the container with id f78b0f392e2f6e00a1d27813d94d898939c4aad49917edbd6b1540223620a798
Sep 05 02:18:55.504068 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:18:55.503926    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ca8378_c8ef_4342_b489_631cba71f8c5.slice/crio-1b21adfd659d58bf4331d46047f4efe9f2e47502b775aef6648ca0df77fa95f3 WatchSource:0}: Error finding container 1b21adfd659d58bf4331d46047f4efe9f2e47502b775aef6648ca0df77fa95f3: Status 404 returned error can't find the container with id 1b21adfd659d58bf4331d46047f4efe9f2e47502b775aef6648ca0df77fa95f3
Sep 05 02:18:56.364895 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:56.364847    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kj85s" event={"ID":"37ca8378-c8ef-4342-b489-631cba71f8c5","Type":"ContainerStarted","Data":"1b21adfd659d58bf4331d46047f4efe9f2e47502b775aef6648ca0df77fa95f3"}
Sep 05 02:18:56.366305 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:56.366276    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-55qfp" event={"ID":"ac1f1f3d-3021-4a29-9162-77a512ca3a29","Type":"ContainerStarted","Data":"f78b0f392e2f6e00a1d27813d94d898939c4aad49917edbd6b1540223620a798"}
Sep 05 02:18:56.610638 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:56.610559    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-tls\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:18:56.610810 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:56.610730    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:56.610810 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:56.610754    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b77fd6b6-n5m6j: secret "image-registry-tls" not found
Sep 05 02:18:56.610948 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:56.610822    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-tls podName:73de4666-9d9c-482a-9f2f-b56dff7dec5b nodeName:}" failed. No retries permitted until 2026-09-05 02:19:00.61080208 +0000 UTC m=+72.090196229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-tls") pod "image-registry-b77fd6b6-n5m6j" (UID: "73de4666-9d9c-482a-9f2f-b56dff7dec5b") : secret "image-registry-tls" not found
Sep 05 02:18:56.999085 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:56.999049    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:57.065540 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.065470    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:57.430989 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.430959    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:57.527450 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.527360    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:57.642529 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.642491    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b2fq5"]
Sep 05 02:18:57.646009 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.645917    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:18:57.660067 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.660019    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 05 02:18:57.660359 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.660341    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 05 02:18:57.661010 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.660988    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7x6sm\""
Sep 05 02:18:57.661104 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.661079    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 05 02:18:57.662104 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.662082    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 05 02:18:57.720068 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.719980    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9czt7\" (UniqueName: \"kubernetes.io/projected/a41caf21-6b3f-4804-a45a-ed0ebec12315-kube-api-access-9czt7\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:18:57.720068 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.720043    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a41caf21-6b3f-4804-a45a-ed0ebec12315-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:18:57.720258 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.720090    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a41caf21-6b3f-4804-a45a-ed0ebec12315-data-volume\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:18:57.720258 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.720133    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a41caf21-6b3f-4804-a45a-ed0ebec12315-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:18:57.720258 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.720179    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a41caf21-6b3f-4804-a45a-ed0ebec12315-crio-socket\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:18:57.752190 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.752159    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b2fq5"]
Sep 05 02:18:57.821478 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.821444    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a41caf21-6b3f-4804-a45a-ed0ebec12315-crio-socket\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:18:57.821646 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.821520    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9czt7\" (UniqueName: \"kubernetes.io/projected/a41caf21-6b3f-4804-a45a-ed0ebec12315-kube-api-access-9czt7\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:18:57.821646 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.821557    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a41caf21-6b3f-4804-a45a-ed0ebec12315-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:18:57.821646 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.821591    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a41caf21-6b3f-4804-a45a-ed0ebec12315-data-volume\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:18:57.821646 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.821626    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a41caf21-6b3f-4804-a45a-ed0ebec12315-crio-socket\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:18:57.821880 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.821637    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a41caf21-6b3f-4804-a45a-ed0ebec12315-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:18:57.821880 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:57.821730    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:57.821880 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:57.821748    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:18:57.821880 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:57.821797    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a41caf21-6b3f-4804-a45a-ed0ebec12315-insights-runtime-extractor-tls podName:a41caf21-6b3f-4804-a45a-ed0ebec12315 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:58.321776598 +0000 UTC m=+69.801170739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a41caf21-6b3f-4804-a45a-ed0ebec12315-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b2fq5" (UID: "a41caf21-6b3f-4804-a45a-ed0ebec12315") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:57.822099 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.821967    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a41caf21-6b3f-4804-a45a-ed0ebec12315-data-volume\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:18:57.822099 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:57.822084    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a41caf21-6b3f-4804-a45a-ed0ebec12315-kube-rbac-proxy-cm podName:a41caf21-6b3f-4804-a45a-ed0ebec12315 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:58.322063835 +0000 UTC m=+69.801457977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a41caf21-6b3f-4804-a45a-ed0ebec12315-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b2fq5" (UID: "a41caf21-6b3f-4804-a45a-ed0ebec12315") : configmap "kube-rbac-proxy" not found
Sep 05 02:18:57.835052 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:57.835018    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9czt7\" (UniqueName: \"kubernetes.io/projected/a41caf21-6b3f-4804-a45a-ed0ebec12315-kube-api-access-9czt7\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:18:58.219957 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:58.219923    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s54l4"]
Sep 05 02:18:58.221691 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:58.221672    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s54l4"
Sep 05 02:18:58.224565 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:58.224539    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 05 02:18:58.224985 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:58.224959    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5dbcz\""
Sep 05 02:18:58.225087 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:58.225074    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 05 02:18:58.232252 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:58.232227    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s54l4"]
Sep 05 02:18:58.325973 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:58.325938    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4tc5\" (UniqueName: \"kubernetes.io/projected/f68966fc-2067-46e7-a102-aa8af80ad1d7-kube-api-access-p4tc5\") pod \"migrator-fdbb8fdff-s54l4\" (UID: \"f68966fc-2067-46e7-a102-aa8af80ad1d7\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s54l4"
Sep 05 02:18:58.326163 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:58.326032    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a41caf21-6b3f-4804-a45a-ed0ebec12315-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:18:58.326163 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:58.326111    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a41caf21-6b3f-4804-a45a-ed0ebec12315-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:18:58.326316 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:58.326299    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:18:58.326389 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:58.326377    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a41caf21-6b3f-4804-a45a-ed0ebec12315-kube-rbac-proxy-cm podName:a41caf21-6b3f-4804-a45a-ed0ebec12315 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:59.326348569 +0000 UTC m=+70.805742709 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a41caf21-6b3f-4804-a45a-ed0ebec12315-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b2fq5" (UID: "a41caf21-6b3f-4804-a45a-ed0ebec12315") : configmap "kube-rbac-proxy" not found
Sep 05 02:18:58.326823 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:58.326799    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:58.326950 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:58.326893    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a41caf21-6b3f-4804-a45a-ed0ebec12315-insights-runtime-extractor-tls podName:a41caf21-6b3f-4804-a45a-ed0ebec12315 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:59.326867313 +0000 UTC m=+70.806261452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a41caf21-6b3f-4804-a45a-ed0ebec12315-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b2fq5" (UID: "a41caf21-6b3f-4804-a45a-ed0ebec12315") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:58.426584 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:58.426544    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4tc5\" (UniqueName: \"kubernetes.io/projected/f68966fc-2067-46e7-a102-aa8af80ad1d7-kube-api-access-p4tc5\") pod \"migrator-fdbb8fdff-s54l4\" (UID: \"f68966fc-2067-46e7-a102-aa8af80ad1d7\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s54l4"
Sep 05 02:18:58.426819 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:58.426636    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:58.435176 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:58.435143    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4tc5\" (UniqueName: \"kubernetes.io/projected/f68966fc-2067-46e7-a102-aa8af80ad1d7-kube-api-access-p4tc5\") pod \"migrator-fdbb8fdff-s54l4\" (UID: \"f68966fc-2067-46e7-a102-aa8af80ad1d7\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s54l4"
Sep 05 02:18:58.492578 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:58.492456    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:58.531682 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:58.531643    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s54l4"
Sep 05 02:18:59.303489 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:59.303457    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-rhwsr"]
Sep 05 02:18:59.307068 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:59.307044    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-rhwsr"
Sep 05 02:18:59.309525 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:59.309503    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 05 02:18:59.309650 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:59.309597    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 05 02:18:59.310358 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:59.310328    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xcbw8\""
Sep 05 02:18:59.310459 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:59.310335    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 05 02:18:59.310459 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:59.310425    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 05 02:18:59.317386 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:59.317359    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-rhwsr"]
Sep 05 02:18:59.333897 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:59.333870    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a41caf21-6b3f-4804-a45a-ed0ebec12315-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:18:59.334013 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:59.333969    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:18:59.334075 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:59.334046    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a41caf21-6b3f-4804-a45a-ed0ebec12315-kube-rbac-proxy-cm podName:a41caf21-6b3f-4804-a45a-ed0ebec12315 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:01.334025256 +0000 UTC m=+72.813419392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a41caf21-6b3f-4804-a45a-ed0ebec12315-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b2fq5" (UID: "a41caf21-6b3f-4804-a45a-ed0ebec12315") : configmap "kube-rbac-proxy" not found
Sep 05 02:18:59.334145 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:59.334074    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/31f1b660-4829-43f4-8006-682ad1321bfb-signing-cabundle\") pod \"service-ca-75d99c484-rhwsr\" (UID: \"31f1b660-4829-43f4-8006-682ad1321bfb\") " pod="openshift-service-ca/service-ca-75d99c484-rhwsr"
Sep 05 02:18:59.334145 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:59.334111    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a41caf21-6b3f-4804-a45a-ed0ebec12315-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:18:59.334145 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:59.334129    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/31f1b660-4829-43f4-8006-682ad1321bfb-signing-key\") pod \"service-ca-75d99c484-rhwsr\" (UID: \"31f1b660-4829-43f4-8006-682ad1321bfb\") " pod="openshift-service-ca/service-ca-75d99c484-rhwsr"
Sep 05 02:18:59.334295 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:59.334145    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9v9c\" (UniqueName: \"kubernetes.io/projected/31f1b660-4829-43f4-8006-682ad1321bfb-kube-api-access-p9v9c\") pod \"service-ca-75d99c484-rhwsr\" (UID: \"31f1b660-4829-43f4-8006-682ad1321bfb\") " pod="openshift-service-ca/service-ca-75d99c484-rhwsr"
Sep 05 02:18:59.334295 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:59.334242    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:59.334295 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:18:59.334294    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a41caf21-6b3f-4804-a45a-ed0ebec12315-insights-runtime-extractor-tls podName:a41caf21-6b3f-4804-a45a-ed0ebec12315 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:01.334279291 +0000 UTC m=+72.813673444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a41caf21-6b3f-4804-a45a-ed0ebec12315-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b2fq5" (UID: "a41caf21-6b3f-4804-a45a-ed0ebec12315") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:59.435234 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:59.435189    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/31f1b660-4829-43f4-8006-682ad1321bfb-signing-cabundle\") pod \"service-ca-75d99c484-rhwsr\" (UID: \"31f1b660-4829-43f4-8006-682ad1321bfb\") " pod="openshift-service-ca/service-ca-75d99c484-rhwsr"
Sep 05 02:18:59.435661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:59.435254    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/31f1b660-4829-43f4-8006-682ad1321bfb-signing-key\") pod \"service-ca-75d99c484-rhwsr\" (UID: \"31f1b660-4829-43f4-8006-682ad1321bfb\") " pod="openshift-service-ca/service-ca-75d99c484-rhwsr"
Sep 05 02:18:59.435661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:59.435280    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9v9c\" (UniqueName: \"kubernetes.io/projected/31f1b660-4829-43f4-8006-682ad1321bfb-kube-api-access-p9v9c\") pod \"service-ca-75d99c484-rhwsr\" (UID: \"31f1b660-4829-43f4-8006-682ad1321bfb\") " pod="openshift-service-ca/service-ca-75d99c484-rhwsr"
Sep 05 02:18:59.436020 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:59.435992    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/31f1b660-4829-43f4-8006-682ad1321bfb-signing-cabundle\") pod \"service-ca-75d99c484-rhwsr\" (UID: \"31f1b660-4829-43f4-8006-682ad1321bfb\") " pod="openshift-service-ca/service-ca-75d99c484-rhwsr"
Sep 05 02:18:59.437871 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:59.437852    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/31f1b660-4829-43f4-8006-682ad1321bfb-signing-key\") pod \"service-ca-75d99c484-rhwsr\" (UID: \"31f1b660-4829-43f4-8006-682ad1321bfb\") " pod="openshift-service-ca/service-ca-75d99c484-rhwsr"
Sep 05 02:18:59.444571 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:59.444545    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9v9c\" (UniqueName: \"kubernetes.io/projected/31f1b660-4829-43f4-8006-682ad1321bfb-kube-api-access-p9v9c\") pod \"service-ca-75d99c484-rhwsr\" (UID: \"31f1b660-4829-43f4-8006-682ad1321bfb\") " pod="openshift-service-ca/service-ca-75d99c484-rhwsr"
Sep 05 02:18:59.569373 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:59.569290    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b89gp_24ff133a-c6d2-4cbc-8dd8-a1e662c05baf/dns-node-resolver/0.log"
Sep 05 02:18:59.618954 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:18:59.618921    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-rhwsr"
Sep 05 02:19:00.090885 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:00.090793    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s54l4"]
Sep 05 02:19:00.094760 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:00.094540    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-rhwsr"]
Sep 05 02:19:00.094891 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:19:00.094805    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68966fc_2067_46e7_a102_aa8af80ad1d7.slice/crio-565c7a4c645a86d9bb2cd32b180e067aeffc027b2c49c5ae7d182f7a9ac29a75 WatchSource:0}: Error finding container 565c7a4c645a86d9bb2cd32b180e067aeffc027b2c49c5ae7d182f7a9ac29a75: Status 404 returned error can't find the container with id 565c7a4c645a86d9bb2cd32b180e067aeffc027b2c49c5ae7d182f7a9ac29a75
Sep 05 02:19:00.098964 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:19:00.098929    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f1b660_4829_43f4_8006_682ad1321bfb.slice/crio-671da962b6d13176d17e123e2af6b2cf34f9f7339486d30a718efdee3f67ed2c WatchSource:0}: Error finding container 671da962b6d13176d17e123e2af6b2cf34f9f7339486d30a718efdee3f67ed2c: Status 404 returned error can't find the container with id 671da962b6d13176d17e123e2af6b2cf34f9f7339486d30a718efdee3f67ed2c
Sep 05 02:19:00.270445 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:00.270412    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:00.379305 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:00.379275    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kj85s" event={"ID":"37ca8378-c8ef-4342-b489-631cba71f8c5","Type":"ContainerStarted","Data":"20fdbab862673a739925fe3565928ffbefd5e29b7356c538c976add76e93bab3"}
Sep 05 02:19:00.380559 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:00.380529    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s54l4" event={"ID":"f68966fc-2067-46e7-a102-aa8af80ad1d7","Type":"ContainerStarted","Data":"565c7a4c645a86d9bb2cd32b180e067aeffc027b2c49c5ae7d182f7a9ac29a75"}
Sep 05 02:19:00.382138 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:00.382115    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-55qfp" event={"ID":"ac1f1f3d-3021-4a29-9162-77a512ca3a29","Type":"ContainerStarted","Data":"8afb3573c10576709c2f0fd3ba08275ea22b689d92c35b2ecdebb88f5531f110"}
Sep 05 02:19:00.382266 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:00.382163    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:19:00.383532 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:00.383513    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-rhwsr" event={"ID":"31f1b660-4829-43f4-8006-682ad1321bfb","Type":"ContainerStarted","Data":"46e7203107cf619e9f7d17351bd86440ef327decd478e1fa2f51d97445aba786"}
Sep 05 02:19:00.383614 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:00.383549    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-rhwsr" event={"ID":"31f1b660-4829-43f4-8006-682ad1321bfb","Type":"ContainerStarted","Data":"671da962b6d13176d17e123e2af6b2cf34f9f7339486d30a718efdee3f67ed2c"}
Sep 05 02:19:00.407149 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:00.407099    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kj85s" podStartSLOduration=67.079526771 podStartE2EDuration="1m11.407079748s" totalImagesPullingTime="4.327552977s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:00.404822679 +0000 UTC m=+71.884216837" watchObservedRunningTime="2026-09-05 02:19:00.407079748 +0000 UTC m=+71.886473905"
Sep 05 02:19:00.422694 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:00.422645    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-rhwsr" podStartSLOduration=1.422631241 podStartE2EDuration="1.422631241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:00.421817167 +0000 UTC m=+71.901211325" watchObservedRunningTime="2026-09-05 02:19:00.422631241 +0000 UTC m=+71.902025398"
Sep 05 02:19:00.441720 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:00.441669    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-55qfp" podStartSLOduration=67.126507457 podStartE2EDuration="1m11.441653581s" totalImagesPullingTime="4.315146124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:00.44076869 +0000 UTC m=+71.920162873" watchObservedRunningTime="2026-09-05 02:19:00.441653581 +0000 UTC m=+71.921047768"
Sep 05 02:19:00.571920 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:00.571891    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vxccn_c290fb8c-2fc3-4aff-863a-d8bf1ee41e01/node-ca/0.log"
Sep 05 02:19:00.646610 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:00.646511    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-tls\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:19:00.646748 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:19:00.646653    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:19:00.646748 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:19:00.646669    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b77fd6b6-n5m6j: secret "image-registry-tls" not found
Sep 05 02:19:00.646748 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:19:00.646721    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-tls podName:73de4666-9d9c-482a-9f2f-b56dff7dec5b nodeName:}" failed. No retries permitted until 2026-09-05 02:19:08.646706177 +0000 UTC m=+80.126100318 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-tls") pod "image-registry-b77fd6b6-n5m6j" (UID: "73de4666-9d9c-482a-9f2f-b56dff7dec5b") : secret "image-registry-tls" not found
Sep 05 02:19:01.206037 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:01.205995    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:01.262992 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:01.262961    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:01.352228 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:01.352191    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a41caf21-6b3f-4804-a45a-ed0ebec12315-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:19:01.352939 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:19:01.352487    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:19:01.352939 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:19:01.352569    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a41caf21-6b3f-4804-a45a-ed0ebec12315-kube-rbac-proxy-cm podName:a41caf21-6b3f-4804-a45a-ed0ebec12315 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:05.352547836 +0000 UTC m=+76.831941986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a41caf21-6b3f-4804-a45a-ed0ebec12315-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b2fq5" (UID: "a41caf21-6b3f-4804-a45a-ed0ebec12315") : configmap "kube-rbac-proxy" not found
Sep 05 02:19:01.352939 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:01.352618    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a41caf21-6b3f-4804-a45a-ed0ebec12315-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:19:01.353179 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:19:01.352972    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:01.353179 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:19:01.353149    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a41caf21-6b3f-4804-a45a-ed0ebec12315-insights-runtime-extractor-tls podName:a41caf21-6b3f-4804-a45a-ed0ebec12315 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:05.353131334 +0000 UTC m=+76.832525474 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a41caf21-6b3f-4804-a45a-ed0ebec12315-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b2fq5" (UID: "a41caf21-6b3f-4804-a45a-ed0ebec12315") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:01.373509 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:01.373478    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:01.470117 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:01.470039    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:01.572643 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:01.572608    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zgfdl_6b81f383-6a65-49a2-a8d8-c5efe33ffe3d/serve-healthcheck-canary/0.log"
Sep 05 02:19:01.603549 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:01.603512    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:01.668265 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:01.668202    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:02.127546 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:02.127507    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:02.392668 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:02.392635    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s54l4" event={"ID":"f68966fc-2067-46e7-a102-aa8af80ad1d7","Type":"ContainerStarted","Data":"0da0cbc581c4076d548f942a36e449137f981da1546c3330bf552e4bc3c4b65b"}
Sep 05 02:19:02.392843 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:02.392686    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s54l4" event={"ID":"f68966fc-2067-46e7-a102-aa8af80ad1d7","Type":"ContainerStarted","Data":"be489e5c37b16f1641be08656b422e7eae38b8e7f3bbc9e37dd35f54a828565f"}
Sep 05 02:19:02.410407 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:02.410364    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s54l4" podStartSLOduration=2.529260423 podStartE2EDuration="4.410350014s" totalImagesPullingTime="1.881089591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:02.409596948 +0000 UTC m=+73.888991108" watchObservedRunningTime="2026-09-05 02:19:02.410350014 +0000 UTC m=+73.889744172"
Sep 05 02:19:02.953009 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:02.952977    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:04.004760 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:04.004679    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:04.076669 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:04.076642    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:04.908201 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:04.908171    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:04.974248 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:04.974216    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:05.122147 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:05.122116    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:05.386899 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:05.386871    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a41caf21-6b3f-4804-a45a-ed0ebec12315-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:19:05.387169 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:05.386936    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a41caf21-6b3f-4804-a45a-ed0ebec12315-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:19:05.390726 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:19:05.387388    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:19:05.390726 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:19:05.387461    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:05.390726 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:19:05.387482    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a41caf21-6b3f-4804-a45a-ed0ebec12315-kube-rbac-proxy-cm podName:a41caf21-6b3f-4804-a45a-ed0ebec12315 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:13.387450888 +0000 UTC m=+84.866845091 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a41caf21-6b3f-4804-a45a-ed0ebec12315-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b2fq5" (UID: "a41caf21-6b3f-4804-a45a-ed0ebec12315") : configmap "kube-rbac-proxy" not found
Sep 05 02:19:05.390726 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:19:05.387536    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a41caf21-6b3f-4804-a45a-ed0ebec12315-insights-runtime-extractor-tls podName:a41caf21-6b3f-4804-a45a-ed0ebec12315 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:13.387517765 +0000 UTC m=+84.866911907 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a41caf21-6b3f-4804-a45a-ed0ebec12315-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b2fq5" (UID: "a41caf21-6b3f-4804-a45a-ed0ebec12315") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:05.530953 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:05.530916    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:06.821987 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:06.821951    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:06.922100 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:06.922072    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:06.985385 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:06.985358    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:07.336113 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:07.336081    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:07.529028 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:07.529000    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:07.595448 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:07.595357    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:07.736446 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:07.736417    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:07.923025 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:07.922994    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:08.714129 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:08.714078    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-tls\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:19:08.716455 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:08.716427    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-tls\") pod \"image-registry-b77fd6b6-n5m6j\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") " pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:19:08.872781 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:08.872735    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:19:08.944105 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:08.944009    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:09.138790 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:09.138737    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:09.154915 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:09.154887    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b77fd6b6-n5m6j"]
Sep 05 02:19:09.158615 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:19:09.158579    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73de4666_9d9c_482a_9f2f_b56dff7dec5b.slice/crio-f35665f2f370ee693d6f1eb9974d3ec1a6fa7f7e13fd889e19f0a424c6623bcc WatchSource:0}: Error finding container f35665f2f370ee693d6f1eb9974d3ec1a6fa7f7e13fd889e19f0a424c6623bcc: Status 404 returned error can't find the container with id f35665f2f370ee693d6f1eb9974d3ec1a6fa7f7e13fd889e19f0a424c6623bcc
Sep 05 02:19:09.327140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:09.327105    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:09.416875 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:09.416783    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j" event={"ID":"73de4666-9d9c-482a-9f2f-b56dff7dec5b","Type":"ContainerStarted","Data":"069317fcf06997da2fc498ef379745132ce7d713944baccd1395c7312e598d95"}
Sep 05 02:19:09.416875 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:09.416853    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j" event={"ID":"73de4666-9d9c-482a-9f2f-b56dff7dec5b","Type":"ContainerStarted","Data":"f35665f2f370ee693d6f1eb9974d3ec1a6fa7f7e13fd889e19f0a424c6623bcc"}
Sep 05 02:19:09.417060 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:09.416880    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:19:09.436901 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:09.436847    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j" podStartSLOduration=17.436813766 podStartE2EDuration="17.436813766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:09.436417942 +0000 UTC m=+80.915812101" watchObservedRunningTime="2026-09-05 02:19:09.436813766 +0000 UTC m=+80.916207924"
Sep 05 02:19:09.562055 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:09.562015    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:10.331093 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:10.331057    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:10.731675 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:10.731641    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:10.904502 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:10.904468    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:11.124984 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:11.124956    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:11.256566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:11.256461    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:11.353597 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:11.353562    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:11.460053 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:11.459969    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:11.748752 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:11.748674    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:11.961264 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:11.961231    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:13.387185 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:13.387154    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:13.454354 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:13.454321    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a41caf21-6b3f-4804-a45a-ed0ebec12315-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:19:13.454521 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:13.454369    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a41caf21-6b3f-4804-a45a-ed0ebec12315-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:19:13.454880 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:13.454863    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a41caf21-6b3f-4804-a45a-ed0ebec12315-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:19:13.456635 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:13.456609    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a41caf21-6b3f-4804-a45a-ed0ebec12315-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b2fq5\" (UID: \"a41caf21-6b3f-4804-a45a-ed0ebec12315\") " pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:19:13.557642 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:13.557609    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b2fq5"
Sep 05 02:19:13.853097 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:13.853074    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b2fq5"]
Sep 05 02:19:13.854910 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:19:13.854886    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41caf21_6b3f_4804_a45a_ed0ebec12315.slice/crio-59f875eef421b592f18b52371c4aaf78d3decd67dca24946a681852f3fdd5e2d WatchSource:0}: Error finding container 59f875eef421b592f18b52371c4aaf78d3decd67dca24946a681852f3fdd5e2d: Status 404 returned error can't find the container with id 59f875eef421b592f18b52371c4aaf78d3decd67dca24946a681852f3fdd5e2d
Sep 05 02:19:14.432775 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:14.432733    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b2fq5" event={"ID":"a41caf21-6b3f-4804-a45a-ed0ebec12315","Type":"ContainerStarted","Data":"3ce0bf17d5cdfb6717d3a70e2bc2cb85414f665671a02841699531155a6659ae"}
Sep 05 02:19:14.433159 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:14.432794    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b2fq5" event={"ID":"a41caf21-6b3f-4804-a45a-ed0ebec12315","Type":"ContainerStarted","Data":"59f875eef421b592f18b52371c4aaf78d3decd67dca24946a681852f3fdd5e2d"}
Sep 05 02:19:14.665996 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:14.665967    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:16.446358 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:16.446298    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b2fq5" event={"ID":"a41caf21-6b3f-4804-a45a-ed0ebec12315","Type":"ContainerStarted","Data":"b1f926b229c188b796275fa6a8719eea3268ed92c9b34843e79c5eb420658acf"}
Sep 05 02:19:17.004087 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:17.004049    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:17.104110 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:17.104074    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:17.451135 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:17.451094    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b2fq5" event={"ID":"a41caf21-6b3f-4804-a45a-ed0ebec12315","Type":"ContainerStarted","Data":"7747228cc895b97b56e5044a01cfdceea8d914b0238cda8318ac8574c7563aa9"}
Sep 05 02:19:17.471382 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:17.471336    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-b2fq5" podStartSLOduration=17.19617615 podStartE2EDuration="20.471320976s" totalImagesPullingTime="3.275144826s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:17.470558572 +0000 UTC m=+88.949952731" watchObservedRunningTime="2026-09-05 02:19:17.471320976 +0000 UTC m=+88.950715134"
Sep 05 02:19:22.706541 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.706505    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"]
Sep 05 02:19:22.709854 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.709824    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"
Sep 05 02:19:22.712458 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.712432    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 05 02:19:22.712580 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.712531    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 05 02:19:22.713281 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.713259    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 05 02:19:22.713374 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.713259    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 05 02:19:22.713374 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.713314    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 05 02:19:22.713374 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.713360    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 05 02:19:22.713534 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.713408    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 05 02:19:22.723334 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.723310    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"]
Sep 05 02:19:22.773156 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.773118    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b77fd6b6-n5m6j"]
Sep 05 02:19:22.822965 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.822929    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37887eef-d2a3-4766-b4c5-60402fee6736-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c8cd57997-7c7q6\" (UID: \"37887eef-d2a3-4766-b4c5-60402fee6736\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"
Sep 05 02:19:22.822965 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.822967    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/37887eef-d2a3-4766-b4c5-60402fee6736-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c8cd57997-7c7q6\" (UID: \"37887eef-d2a3-4766-b4c5-60402fee6736\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"
Sep 05 02:19:22.823201 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.823061    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcbm6\" (UniqueName: \"kubernetes.io/projected/37887eef-d2a3-4766-b4c5-60402fee6736-kube-api-access-kcbm6\") pod \"cluster-proxy-proxy-agent-5c8cd57997-7c7q6\" (UID: \"37887eef-d2a3-4766-b4c5-60402fee6736\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"
Sep 05 02:19:22.823201 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.823098    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/37887eef-d2a3-4766-b4c5-60402fee6736-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c8cd57997-7c7q6\" (UID: \"37887eef-d2a3-4766-b4c5-60402fee6736\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"
Sep 05 02:19:22.823305 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.823196    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/37887eef-d2a3-4766-b4c5-60402fee6736-hub\") pod \"cluster-proxy-proxy-agent-5c8cd57997-7c7q6\" (UID: \"37887eef-d2a3-4766-b4c5-60402fee6736\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"
Sep 05 02:19:22.823305 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.823228    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/37887eef-d2a3-4766-b4c5-60402fee6736-ca\") pod \"cluster-proxy-proxy-agent-5c8cd57997-7c7q6\" (UID: \"37887eef-d2a3-4766-b4c5-60402fee6736\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"
Sep 05 02:19:22.870708 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.870672    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:22.923574 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.923538    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/37887eef-d2a3-4766-b4c5-60402fee6736-hub\") pod \"cluster-proxy-proxy-agent-5c8cd57997-7c7q6\" (UID: \"37887eef-d2a3-4766-b4c5-60402fee6736\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"
Sep 05 02:19:22.923574 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.923572    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/37887eef-d2a3-4766-b4c5-60402fee6736-ca\") pod \"cluster-proxy-proxy-agent-5c8cd57997-7c7q6\" (UID: \"37887eef-d2a3-4766-b4c5-60402fee6736\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"
Sep 05 02:19:22.923791 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.923594    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37887eef-d2a3-4766-b4c5-60402fee6736-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c8cd57997-7c7q6\" (UID: \"37887eef-d2a3-4766-b4c5-60402fee6736\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"
Sep 05 02:19:22.923791 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.923614    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/37887eef-d2a3-4766-b4c5-60402fee6736-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c8cd57997-7c7q6\" (UID: \"37887eef-d2a3-4766-b4c5-60402fee6736\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"
Sep 05 02:19:22.923791 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.923654    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcbm6\" (UniqueName: \"kubernetes.io/projected/37887eef-d2a3-4766-b4c5-60402fee6736-kube-api-access-kcbm6\") pod \"cluster-proxy-proxy-agent-5c8cd57997-7c7q6\" (UID: \"37887eef-d2a3-4766-b4c5-60402fee6736\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"
Sep 05 02:19:22.923791 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.923689    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/37887eef-d2a3-4766-b4c5-60402fee6736-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c8cd57997-7c7q6\" (UID: \"37887eef-d2a3-4766-b4c5-60402fee6736\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"
Sep 05 02:19:22.924426 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.924399    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/37887eef-d2a3-4766-b4c5-60402fee6736-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c8cd57997-7c7q6\" (UID: \"37887eef-d2a3-4766-b4c5-60402fee6736\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"
Sep 05 02:19:22.926123 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.926102    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/37887eef-d2a3-4766-b4c5-60402fee6736-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c8cd57997-7c7q6\" (UID: \"37887eef-d2a3-4766-b4c5-60402fee6736\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"
Sep 05 02:19:22.926312 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.926290    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37887eef-d2a3-4766-b4c5-60402fee6736-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c8cd57997-7c7q6\" (UID: \"37887eef-d2a3-4766-b4c5-60402fee6736\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"
Sep 05 02:19:22.926420 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.926404    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/37887eef-d2a3-4766-b4c5-60402fee6736-hub\") pod \"cluster-proxy-proxy-agent-5c8cd57997-7c7q6\" (UID: \"37887eef-d2a3-4766-b4c5-60402fee6736\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"
Sep 05 02:19:22.926482 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.926418    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/37887eef-d2a3-4766-b4c5-60402fee6736-ca\") pod \"cluster-proxy-proxy-agent-5c8cd57997-7c7q6\" (UID: \"37887eef-d2a3-4766-b4c5-60402fee6736\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"
Sep 05 02:19:22.934771 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.934751    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcbm6\" (UniqueName: \"kubernetes.io/projected/37887eef-d2a3-4766-b4c5-60402fee6736-kube-api-access-kcbm6\") pod \"cluster-proxy-proxy-agent-5c8cd57997-7c7q6\" (UID: \"37887eef-d2a3-4766-b4c5-60402fee6736\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"
Sep 05 02:19:22.967594 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:22.967537    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:23.029527 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:23.029496    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"
Sep 05 02:19:23.305910 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:23.305821    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6"]
Sep 05 02:19:23.309602 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:19:23.309577    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37887eef_d2a3_4766_b4c5_60402fee6736.slice/crio-ec061832502df81b03cafa2dee5a92f1aeabb068810174bfa9d2d85632f32d35 WatchSource:0}: Error finding container ec061832502df81b03cafa2dee5a92f1aeabb068810174bfa9d2d85632f32d35: Status 404 returned error can't find the container with id ec061832502df81b03cafa2dee5a92f1aeabb068810174bfa9d2d85632f32d35
Sep 05 02:19:23.470092 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:23.470053    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6" event={"ID":"37887eef-d2a3-4766-b4c5-60402fee6736","Type":"ContainerStarted","Data":"ec061832502df81b03cafa2dee5a92f1aeabb068810174bfa9d2d85632f32d35"}
Sep 05 02:19:25.632721 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:25.632691    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:25.696656 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:25.696624    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:26.401666 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:26.401635    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:26.502393 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:26.502358    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:26.652679 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:26.652594    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls\") pod \"dns-default-66mtm\" (UID: \"18051063-ebe9-4362-ab51-4a4da32e15ff\") " pod="openshift-dns/dns-default-66mtm"
Sep 05 02:19:26.654821 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:26.654790    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18051063-ebe9-4362-ab51-4a4da32e15ff-metrics-tls\") pod \"dns-default-66mtm\" (UID: \"18051063-ebe9-4362-ab51-4a4da32e15ff\") " pod="openshift-dns/dns-default-66mtm"
Sep 05 02:19:26.707303 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:26.707265    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cl7gt\""
Sep 05 02:19:26.715104 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:26.715079    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-66mtm"
Sep 05 02:19:26.850677 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:26.850647    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:26.990339 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:26.990268    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-66mtm"]
Sep 05 02:19:26.993364 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:19:26.993333    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18051063_ebe9_4362_ab51_4a4da32e15ff.slice/crio-6b186c3734fd36c82c3545a7e8c6fc7864ae8e651a0bfe39fc8013b91506c44e WatchSource:0}: Error finding container 6b186c3734fd36c82c3545a7e8c6fc7864ae8e651a0bfe39fc8013b91506c44e: Status 404 returned error can't find the container with id 6b186c3734fd36c82c3545a7e8c6fc7864ae8e651a0bfe39fc8013b91506c44e
Sep 05 02:19:27.484443 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:27.484386    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-66mtm" event={"ID":"18051063-ebe9-4362-ab51-4a4da32e15ff","Type":"ContainerStarted","Data":"6b186c3734fd36c82c3545a7e8c6fc7864ae8e651a0bfe39fc8013b91506c44e"}
Sep 05 02:19:29.253258 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:29.253225    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:31.390646 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:31.390609    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-55qfp"
Sep 05 02:19:31.502593 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:31.502558    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6" event={"ID":"37887eef-d2a3-4766-b4c5-60402fee6736","Type":"ContainerStarted","Data":"43f607a67ae2c6fdd496c441f16cd54c5028c0baddd35e2132ca951a28505e19"}
Sep 05 02:19:31.502789 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:31.502602    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6" event={"ID":"37887eef-d2a3-4766-b4c5-60402fee6736","Type":"ContainerStarted","Data":"dac264f2f3c66752952880abf178ad9198eb02f813f074f4ed262cdef7193daf"}
Sep 05 02:19:31.502789 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:31.502629    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6" event={"ID":"37887eef-d2a3-4766-b4c5-60402fee6736","Type":"ContainerStarted","Data":"f6d3c04d4511bef96fd4f83c2c6d1456d026afe1595de1455132e202f6a236c0"}
Sep 05 02:19:31.504446 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:31.504418    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-66mtm" event={"ID":"18051063-ebe9-4362-ab51-4a4da32e15ff","Type":"ContainerStarted","Data":"b9d7db9124bb68f2cd12fbb39965ff3573d36baf7fd8e58849b2b61b014cc6c6"}
Sep 05 02:19:31.504552 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:31.504461    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-66mtm" event={"ID":"18051063-ebe9-4362-ab51-4a4da32e15ff","Type":"ContainerStarted","Data":"400fe8aa7a40df05841ab5d45d606f0efe78641ea3b36080e35267889ea045d1"}
Sep 05 02:19:31.504552 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:31.504482    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-66mtm"
Sep 05 02:19:31.540243 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:31.540191    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c8cd57997-7c7q6" podStartSLOduration=2.14628986 podStartE2EDuration="9.540173132s" totalImagesPullingTime="7.393883272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:31.540018524 +0000 UTC m=+103.019412684" watchObservedRunningTime="2026-09-05 02:19:31.540173132 +0000 UTC m=+103.019567285"
Sep 05 02:19:31.593092 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:31.593025    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-66mtm" podStartSLOduration=65.893267018 podStartE2EDuration="1m9.593006212s" totalImagesPullingTime="3.699739194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:31.592846799 +0000 UTC m=+103.072240956" watchObservedRunningTime="2026-09-05 02:19:31.593006212 +0000 UTC m=+103.072400371"
Sep 05 02:19:31.687864 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:31.687752    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:31.882542 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:31.882508    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:32.128663 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.128619    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:32.281089 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.281058    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kw8ds"]
Sep 05 02:19:32.314512 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.314475    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.317977 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.317954    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 05 02:19:32.317977 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.317972    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 05 02:19:32.318527 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.318507    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 05 02:19:32.318588 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.318541    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 05 02:19:32.318912 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.318895    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pzst5\""
Sep 05 02:19:32.319071 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.319057    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 05 02:19:32.319169 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.319154    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 05 02:19:32.355224 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.355182    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:32.400768 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.400670    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-node-exporter-wtmp\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.400768 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.400711    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-node-exporter-tls\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.400768 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.400731    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-metrics-client-ca\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.401234 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.400785    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-sys\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.401234 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.400887    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-node-exporter-accelerators-collector-config\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.401234 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.400909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-root\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.401234 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.400926    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.401234 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.400940    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf58t\" (UniqueName: \"kubernetes.io/projected/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-kube-api-access-xf58t\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.401234 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.400957    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-node-exporter-textfile\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.484471 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.484439    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:32.501601 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.501576    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-node-exporter-accelerators-collector-config\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.501701 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.501607    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-root\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.501701 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.501625    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.501701 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.501643    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf58t\" (UniqueName: \"kubernetes.io/projected/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-kube-api-access-xf58t\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.501701 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.501661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-node-exporter-textfile\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.501701 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.501682    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-node-exporter-wtmp\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.501997 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.501701    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-root\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.501997 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.501821    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-node-exporter-wtmp\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.501997 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.501850    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-node-exporter-tls\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.501997 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.501888    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-metrics-client-ca\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.501997 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.501945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-sys\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.502238 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.502045    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-sys\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.502238 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.502059    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-node-exporter-textfile\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.502319 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.502263    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-node-exporter-accelerators-collector-config\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.502440 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.502414    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-metrics-client-ca\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.504168 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.504140    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-node-exporter-tls\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.504272 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.504193    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.513188 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.513165    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf58t\" (UniqueName: \"kubernetes.io/projected/48cd7ba6-45a0-4508-a0dd-0a04e77ab363-kube-api-access-xf58t\") pod \"node-exporter-kw8ds\" (UID: \"48cd7ba6-45a0-4508-a0dd-0a04e77ab363\") " pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.589881 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.589850    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:32.624477 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.624449    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kw8ds"
Sep 05 02:19:32.657240 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:19:32.657211    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48cd7ba6_45a0_4508_a0dd_0a04e77ab363.slice/crio-720ffa9c7b3f54c0cae58c6d54d9bc508b0075328d3503be7488a2e617c7933d WatchSource:0}: Error finding container 720ffa9c7b3f54c0cae58c6d54d9bc508b0075328d3503be7488a2e617c7933d: Status 404 returned error can't find the container with id 720ffa9c7b3f54c0cae58c6d54d9bc508b0075328d3503be7488a2e617c7933d
Sep 05 02:19:32.778414 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.778375    2566 patch_prober.go:28] interesting pod/image-registry-b77fd6b6-n5m6j container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 05 02:19:32.778572 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.778429    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j" podUID="73de4666-9d9c-482a-9f2f-b56dff7dec5b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:19:32.817878 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.817844    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:32.908055 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:32.907981    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:33.002568 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:33.002534    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:33.118482 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:33.118451    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:33.511918 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:33.511880    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kw8ds" event={"ID":"48cd7ba6-45a0-4508-a0dd-0a04e77ab363","Type":"ContainerStarted","Data":"720ffa9c7b3f54c0cae58c6d54d9bc508b0075328d3503be7488a2e617c7933d"}
Sep 05 02:19:34.516768 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:34.516732    2566 generic.go:356] "Generic (PLEG): container finished" podID="48cd7ba6-45a0-4508-a0dd-0a04e77ab363" containerID="8f1d423e9baac8164ab1f04b472981f2df2909bf8293ea1acbf6cb634daf21a1" exitCode=0
Sep 05 02:19:34.517186 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:34.516779    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kw8ds" event={"ID":"48cd7ba6-45a0-4508-a0dd-0a04e77ab363","Type":"ContainerDied","Data":"8f1d423e9baac8164ab1f04b472981f2df2909bf8293ea1acbf6cb634daf21a1"}
Sep 05 02:19:34.893600 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:34.893561    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:34.965914 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:34.965879    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:35.034781 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:35.034749    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:35.296385 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:35.296292    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:35.404214 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:35.404187    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:35.467407 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:35.467379    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:35.522499 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:35.522469    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kw8ds" event={"ID":"48cd7ba6-45a0-4508-a0dd-0a04e77ab363","Type":"ContainerStarted","Data":"b0c7280a3aafcfc48aee553dce953c249d6b79a4e5a08333811e040e2e3b0970"}
Sep 05 02:19:35.522883 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:35.522515    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kw8ds" event={"ID":"48cd7ba6-45a0-4508-a0dd-0a04e77ab363","Type":"ContainerStarted","Data":"0ba3942f47e23936354bc036f4fb200a9dd4f047dcf2befcab39b4a4d0e697c9"}
Sep 05 02:19:35.543139 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:35.543095    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kw8ds" podStartSLOduration=2.03943554 podStartE2EDuration="3.543081131s" totalImagesPullingTime="1.427623013s" totalInitContainerRuntime="76.022578ms" isStatefulPod=true podCreationTimestamp="2026-09-05 02:19:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:35.542862041 +0000 UTC m=+107.022256200" watchObservedRunningTime="2026-09-05 02:19:35.543081131 +0000 UTC m=+107.022475288"
Sep 05 02:19:36.623999 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:36.623962    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:36.841953 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:36.841922    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:38.240761 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:38.240730    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:39.810819 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:39.810773    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:39.912434 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:39.912392    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:40.237521 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:40.237494    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:40.338504 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:40.338470    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:41.509440 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:41.509410    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-66mtm"
Sep 05 02:19:41.939300 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:41.939269    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:42.047636 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:42.047599    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:42.147324 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:42.147261    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:42.777523 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:42.777496    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:19:44.507752 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:44.507703    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:44.614992 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:44.614956    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:44.777959 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:44.777858    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:44.895732 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:44.895681    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:46.993448 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:46.993410    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:47.060625 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:47.060594    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:47.792692 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:47.792653    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j" podUID="73de4666-9d9c-482a-9f2f-b56dff7dec5b" containerName="registry" containerID="cri-o://069317fcf06997da2fc498ef379745132ce7d713944baccd1395c7312e598d95" gracePeriod=30
Sep 05 02:19:48.028005 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.027982    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:19:48.125260 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.125232    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73de4666-9d9c-482a-9f2f-b56dff7dec5b-installation-pull-secrets\") pod \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") "
Sep 05 02:19:48.125426 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.125287    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-bound-sa-token\") pod \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") "
Sep 05 02:19:48.125426 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.125317    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73de4666-9d9c-482a-9f2f-b56dff7dec5b-ca-trust-extracted\") pod \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") "
Sep 05 02:19:48.125426 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.125336    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73de4666-9d9c-482a-9f2f-b56dff7dec5b-image-registry-private-configuration\") pod \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") "
Sep 05 02:19:48.125426 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.125353    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73de4666-9d9c-482a-9f2f-b56dff7dec5b-trusted-ca\") pod \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") "
Sep 05 02:19:48.125426 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.125380    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-tls\") pod \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") "
Sep 05 02:19:48.125426 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.125400    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-certificates\") pod \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") "
Sep 05 02:19:48.125728 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.125451    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhgf6\" (UniqueName: \"kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-kube-api-access-lhgf6\") pod \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\" (UID: \"73de4666-9d9c-482a-9f2f-b56dff7dec5b\") "
Sep 05 02:19:48.126064 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.125999    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "73de4666-9d9c-482a-9f2f-b56dff7dec5b" (UID: "73de4666-9d9c-482a-9f2f-b56dff7dec5b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:19:48.126197 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.126070    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73de4666-9d9c-482a-9f2f-b56dff7dec5b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "73de4666-9d9c-482a-9f2f-b56dff7dec5b" (UID: "73de4666-9d9c-482a-9f2f-b56dff7dec5b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:19:48.128308 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.128276    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "73de4666-9d9c-482a-9f2f-b56dff7dec5b" (UID: "73de4666-9d9c-482a-9f2f-b56dff7dec5b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:19:48.128423 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.128283    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-kube-api-access-lhgf6" (OuterVolumeSpecName: "kube-api-access-lhgf6") pod "73de4666-9d9c-482a-9f2f-b56dff7dec5b" (UID: "73de4666-9d9c-482a-9f2f-b56dff7dec5b"). InnerVolumeSpecName "kube-api-access-lhgf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:19:48.128423 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.128321    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73de4666-9d9c-482a-9f2f-b56dff7dec5b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "73de4666-9d9c-482a-9f2f-b56dff7dec5b" (UID: "73de4666-9d9c-482a-9f2f-b56dff7dec5b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:19:48.128423 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.128403    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73de4666-9d9c-482a-9f2f-b56dff7dec5b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "73de4666-9d9c-482a-9f2f-b56dff7dec5b" (UID: "73de4666-9d9c-482a-9f2f-b56dff7dec5b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:19:48.128593 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.128449    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "73de4666-9d9c-482a-9f2f-b56dff7dec5b" (UID: "73de4666-9d9c-482a-9f2f-b56dff7dec5b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:19:48.134704 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.134682    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73de4666-9d9c-482a-9f2f-b56dff7dec5b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "73de4666-9d9c-482a-9f2f-b56dff7dec5b" (UID: "73de4666-9d9c-482a-9f2f-b56dff7dec5b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:19:48.226451 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.226412    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-bound-sa-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:19:48.226451 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.226441    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73de4666-9d9c-482a-9f2f-b56dff7dec5b-ca-trust-extracted\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:19:48.226451 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.226451    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73de4666-9d9c-482a-9f2f-b56dff7dec5b-image-registry-private-configuration\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:19:48.226451 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.226463    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73de4666-9d9c-482a-9f2f-b56dff7dec5b-trusted-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:19:48.226752 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.226473    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-tls\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:19:48.226752 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.226480    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73de4666-9d9c-482a-9f2f-b56dff7dec5b-registry-certificates\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:19:48.226752 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.226488    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhgf6\" (UniqueName: \"kubernetes.io/projected/73de4666-9d9c-482a-9f2f-b56dff7dec5b-kube-api-access-lhgf6\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:19:48.226752 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.226496    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73de4666-9d9c-482a-9f2f-b56dff7dec5b-installation-pull-secrets\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:19:48.565265 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.565183    2566 generic.go:356] "Generic (PLEG): container finished" podID="73de4666-9d9c-482a-9f2f-b56dff7dec5b" containerID="069317fcf06997da2fc498ef379745132ce7d713944baccd1395c7312e598d95" exitCode=0
Sep 05 02:19:48.565265 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.565225    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j" event={"ID":"73de4666-9d9c-482a-9f2f-b56dff7dec5b","Type":"ContainerDied","Data":"069317fcf06997da2fc498ef379745132ce7d713944baccd1395c7312e598d95"}
Sep 05 02:19:48.565265 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.565228    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j"
Sep 05 02:19:48.565265 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.565264    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b77fd6b6-n5m6j" event={"ID":"73de4666-9d9c-482a-9f2f-b56dff7dec5b","Type":"ContainerDied","Data":"f35665f2f370ee693d6f1eb9974d3ec1a6fa7f7e13fd889e19f0a424c6623bcc"}
Sep 05 02:19:48.565512 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.565285    2566 scope.go:117] "RemoveContainer" containerID="069317fcf06997da2fc498ef379745132ce7d713944baccd1395c7312e598d95"
Sep 05 02:19:48.573723 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.573705    2566 scope.go:117] "RemoveContainer" containerID="069317fcf06997da2fc498ef379745132ce7d713944baccd1395c7312e598d95"
Sep 05 02:19:48.574001 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:19:48.573980    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"069317fcf06997da2fc498ef379745132ce7d713944baccd1395c7312e598d95\": container with ID starting with 069317fcf06997da2fc498ef379745132ce7d713944baccd1395c7312e598d95 not found: ID does not exist" containerID="069317fcf06997da2fc498ef379745132ce7d713944baccd1395c7312e598d95"
Sep 05 02:19:48.574050 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.574010    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"069317fcf06997da2fc498ef379745132ce7d713944baccd1395c7312e598d95"} err="failed to get container status \"069317fcf06997da2fc498ef379745132ce7d713944baccd1395c7312e598d95\": rpc error: code = NotFound desc = could not find container \"069317fcf06997da2fc498ef379745132ce7d713944baccd1395c7312e598d95\": container with ID starting with 069317fcf06997da2fc498ef379745132ce7d713944baccd1395c7312e598d95 not found: ID does not exist"
Sep 05 02:19:48.587951 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.587930    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b77fd6b6-n5m6j"]
Sep 05 02:19:48.590147 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:48.590125    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b77fd6b6-n5m6j"]
Sep 05 02:19:49.048316 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:49.048284    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73de4666-9d9c-482a-9f2f-b56dff7dec5b" path="/var/lib/kubelet/pods/73de4666-9d9c-482a-9f2f-b56dff7dec5b/volumes"
Sep 05 02:19:50.671643 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:50.671609    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:51.885397 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:51.885365    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:51.985466 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:51.985435    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:52.966451 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:52.966418    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-52z87"]
Sep 05 02:19:52.966857 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:52.966697    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73de4666-9d9c-482a-9f2f-b56dff7dec5b" containerName="registry"
Sep 05 02:19:52.966857 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:52.966709    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="73de4666-9d9c-482a-9f2f-b56dff7dec5b" containerName="registry"
Sep 05 02:19:52.966857 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:52.966753    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="73de4666-9d9c-482a-9f2f-b56dff7dec5b" containerName="registry"
Sep 05 02:19:52.969701 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:52.969684    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-52z87"
Sep 05 02:19:52.973135 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:52.973115    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 05 02:19:52.975261 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:52.975241    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 05 02:19:52.977037 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:52.977021    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-rzjml\""
Sep 05 02:19:53.004191 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:53.004171    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-52z87"]
Sep 05 02:19:53.053638 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:53.053616    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:53.063021 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:53.063001    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf9hm\" (UniqueName: \"kubernetes.io/projected/4e6c1fd5-e86b-497b-b5a3-06616e4fd09b-kube-api-access-qf9hm\") pod \"downloads-85d98c998f-52z87\" (UID: \"4e6c1fd5-e86b-497b-b5a3-06616e4fd09b\") " pod="openshift-console/downloads-85d98c998f-52z87"
Sep 05 02:19:53.164325 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:53.164286    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf9hm\" (UniqueName: \"kubernetes.io/projected/4e6c1fd5-e86b-497b-b5a3-06616e4fd09b-kube-api-access-qf9hm\") pod \"downloads-85d98c998f-52z87\" (UID: \"4e6c1fd5-e86b-497b-b5a3-06616e4fd09b\") " pod="openshift-console/downloads-85d98c998f-52z87"
Sep 05 02:19:53.173057 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:53.173025    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf9hm\" (UniqueName: \"kubernetes.io/projected/4e6c1fd5-e86b-497b-b5a3-06616e4fd09b-kube-api-access-qf9hm\") pod \"downloads-85d98c998f-52z87\" (UID: \"4e6c1fd5-e86b-497b-b5a3-06616e4fd09b\") " pod="openshift-console/downloads-85d98c998f-52z87"
Sep 05 02:19:53.278166 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:53.278084    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-52z87"
Sep 05 02:19:53.551261 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:53.551237    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-52z87"]
Sep 05 02:19:53.553378 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:19:53.553342    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6c1fd5_e86b_497b_b5a3_06616e4fd09b.slice/crio-9dcb61c40bd3c0603b04701bd1d558248910a1439759463e7669129bd7eb8684 WatchSource:0}: Error finding container 9dcb61c40bd3c0603b04701bd1d558248910a1439759463e7669129bd7eb8684: Status 404 returned error can't find the container with id 9dcb61c40bd3c0603b04701bd1d558248910a1439759463e7669129bd7eb8684
Sep 05 02:19:53.581587 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:53.581554    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-52z87" event={"ID":"4e6c1fd5-e86b-497b-b5a3-06616e4fd09b","Type":"ContainerStarted","Data":"9dcb61c40bd3c0603b04701bd1d558248910a1439759463e7669129bd7eb8684"}
Sep 05 02:19:54.098608 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:54.098571    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:54.172441 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:54.172407    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:54.648797 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:54.648764    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:54.843110 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:54.843063    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:54.943548 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:54.943457    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:55.672908 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:55.672868    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:55.777787 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:55.777751    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:58.811030 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:58.810989    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs\") pod \"network-metrics-daemon-262h2\" (UID: \"e701e309-b8b1-47c5-927a-ce15959fafa1\") " pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:19:58.813462 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:58.813433    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e701e309-b8b1-47c5-927a-ce15959fafa1-metrics-certs\") pod \"network-metrics-daemon-262h2\" (UID: \"e701e309-b8b1-47c5-927a-ce15959fafa1\") " pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:19:59.055790 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.055764    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4nzpg\""
Sep 05 02:19:59.063993 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.063936    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-262h2"
Sep 05 02:19:59.342232 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.342207    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-262h2"]
Sep 05 02:19:59.344500 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:19:59.344470    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode701e309_b8b1_47c5_927a_ce15959fafa1.slice/crio-0382317f5d17369985537950bd6c6b5ff65738745a32698997d975a4190ce940 WatchSource:0}: Error finding container 0382317f5d17369985537950bd6c6b5ff65738745a32698997d975a4190ce940: Status 404 returned error can't find the container with id 0382317f5d17369985537950bd6c6b5ff65738745a32698997d975a4190ce940
Sep 05 02:19:59.575106 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.575062    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-747844c66d-vdz6g"]
Sep 05 02:19:59.577774 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.577754    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:19:59.580002 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.579966    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 05 02:19:59.580114 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.580093    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 05 02:19:59.580241 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.580225    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 05 02:19:59.580878 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.580856    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 05 02:19:59.580878 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.580873    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 05 02:19:59.581029 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.580858    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4hkzj\""
Sep 05 02:19:59.590227 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.590204    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-747844c66d-vdz6g"]
Sep 05 02:19:59.608983 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.608920    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-262h2" event={"ID":"e701e309-b8b1-47c5-927a-ce15959fafa1","Type":"ContainerStarted","Data":"0382317f5d17369985537950bd6c6b5ff65738745a32698997d975a4190ce940"}
Sep 05 02:19:59.617817 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.617787    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/196203dc-fbce-4fc0-8835-19690ca7186d-console-oauth-config\") pod \"console-747844c66d-vdz6g\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") " pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:19:59.617931 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.617825    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/196203dc-fbce-4fc0-8835-19690ca7186d-console-serving-cert\") pod \"console-747844c66d-vdz6g\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") " pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:19:59.617931 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.617920    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzvrp\" (UniqueName: \"kubernetes.io/projected/196203dc-fbce-4fc0-8835-19690ca7186d-kube-api-access-wzvrp\") pod \"console-747844c66d-vdz6g\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") " pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:19:59.618056 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.617958    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/196203dc-fbce-4fc0-8835-19690ca7186d-oauth-serving-cert\") pod \"console-747844c66d-vdz6g\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") " pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:19:59.618056 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.617991    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/196203dc-fbce-4fc0-8835-19690ca7186d-console-config\") pod \"console-747844c66d-vdz6g\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") " pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:19:59.618056 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.618020    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/196203dc-fbce-4fc0-8835-19690ca7186d-service-ca\") pod \"console-747844c66d-vdz6g\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") " pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:19:59.696018 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.695989    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:59.719300 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.719271    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/196203dc-fbce-4fc0-8835-19690ca7186d-console-oauth-config\") pod \"console-747844c66d-vdz6g\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") " pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:19:59.719426 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.719305    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/196203dc-fbce-4fc0-8835-19690ca7186d-console-serving-cert\") pod \"console-747844c66d-vdz6g\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") " pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:19:59.719426 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.719342    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzvrp\" (UniqueName: \"kubernetes.io/projected/196203dc-fbce-4fc0-8835-19690ca7186d-kube-api-access-wzvrp\") pod \"console-747844c66d-vdz6g\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") " pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:19:59.719426 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.719371    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/196203dc-fbce-4fc0-8835-19690ca7186d-oauth-serving-cert\") pod \"console-747844c66d-vdz6g\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") " pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:19:59.719426 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.719398    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/196203dc-fbce-4fc0-8835-19690ca7186d-console-config\") pod \"console-747844c66d-vdz6g\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") " pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:19:59.719638 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.719427    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/196203dc-fbce-4fc0-8835-19690ca7186d-service-ca\") pod \"console-747844c66d-vdz6g\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") " pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:19:59.720237 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.720209    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/196203dc-fbce-4fc0-8835-19690ca7186d-service-ca\") pod \"console-747844c66d-vdz6g\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") " pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:19:59.720338 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.720259    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/196203dc-fbce-4fc0-8835-19690ca7186d-oauth-serving-cert\") pod \"console-747844c66d-vdz6g\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") " pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:19:59.720338 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.720310    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/196203dc-fbce-4fc0-8835-19690ca7186d-console-config\") pod \"console-747844c66d-vdz6g\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") " pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:19:59.722084 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.722058    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/196203dc-fbce-4fc0-8835-19690ca7186d-console-oauth-config\") pod \"console-747844c66d-vdz6g\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") " pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:19:59.722193 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.722169    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/196203dc-fbce-4fc0-8835-19690ca7186d-console-serving-cert\") pod \"console-747844c66d-vdz6g\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") " pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:19:59.727532 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.727514    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzvrp\" (UniqueName: \"kubernetes.io/projected/196203dc-fbce-4fc0-8835-19690ca7186d-kube-api-access-wzvrp\") pod \"console-747844c66d-vdz6g\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") " pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:19:59.888431 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:19:59.888398    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:20:00.171929 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:00.171849    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-747844c66d-vdz6g"]
Sep 05 02:20:00.175442 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:20:00.175410    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod196203dc_fbce_4fc0_8835_19690ca7186d.slice/crio-d91fe946a16e3b4db890de66389e6ec5e513d7d349b51117f1c209049e8772d7 WatchSource:0}: Error finding container d91fe946a16e3b4db890de66389e6ec5e513d7d349b51117f1c209049e8772d7: Status 404 returned error can't find the container with id d91fe946a16e3b4db890de66389e6ec5e513d7d349b51117f1c209049e8772d7
Sep 05 02:20:00.615032 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:00.614948    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747844c66d-vdz6g" event={"ID":"196203dc-fbce-4fc0-8835-19690ca7186d","Type":"ContainerStarted","Data":"d91fe946a16e3b4db890de66389e6ec5e513d7d349b51117f1c209049e8772d7"}
Sep 05 02:20:02.047868 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:02.047816    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:02.108175 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:02.108117    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:02.624983 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:02.624932    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-262h2" event={"ID":"e701e309-b8b1-47c5-927a-ce15959fafa1","Type":"ContainerStarted","Data":"b8f80a10a421bd208bb62d60de637d6ec70327763733784a973cbc366853f102"}
Sep 05 02:20:02.625169 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:02.625001    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-262h2" event={"ID":"e701e309-b8b1-47c5-927a-ce15959fafa1","Type":"ContainerStarted","Data":"f2c69d19dc13f97efdcbc037279e85a8441bd07e77d46ccfb7b5348bebd79470"}
Sep 05 02:20:02.643703 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:02.642692    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-262h2" podStartSLOduration=131.26066162 podStartE2EDuration="2m13.642672965s" totalImagesPullingTime="2.382011345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:02.64234755 +0000 UTC m=+134.121741717" watchObservedRunningTime="2026-09-05 02:20:02.642672965 +0000 UTC m=+134.122067124"
Sep 05 02:20:02.740121 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:02.740081    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:02.806136 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:02.806105    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:06.683980 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:06.683943    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:06.784305 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:06.784262    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:09.031982 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:09.031948    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:09.159074 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:09.159036    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:10.082135 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:10.082098    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:10.657500 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:10.657464    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747844c66d-vdz6g" event={"ID":"196203dc-fbce-4fc0-8835-19690ca7186d","Type":"ContainerStarted","Data":"01e89b995b125d38ed172ad1e3d95ae9462a347cdde1c33d2b991e1647ca3bc4"}
Sep 05 02:20:10.659713 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:10.659257    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-52z87" event={"ID":"4e6c1fd5-e86b-497b-b5a3-06616e4fd09b","Type":"ContainerStarted","Data":"313aaaf551bd4964ab0654031138765755ac49312a3d4617a13711145f68dd86"}
Sep 05 02:20:10.659713 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:10.659601    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-52z87"
Sep 05 02:20:10.661086 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:10.661065    2566 patch_prober.go:28] interesting pod/downloads-85d98c998f-52z87 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.14:8080/\": dial tcp 10.133.0.14:8080: connect: connection refused" start-of-body=
Sep 05 02:20:10.661177 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:10.661102    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-85d98c998f-52z87" podUID="4e6c1fd5-e86b-497b-b5a3-06616e4fd09b" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.14:8080/\": dial tcp 10.133.0.14:8080: connect: connection refused"
Sep 05 02:20:10.682846 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:10.682785    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-747844c66d-vdz6g" podStartSLOduration=1.300619621 podStartE2EDuration="11.682770397s" totalImagesPullingTime="10.382150776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:10.680399498 +0000 UTC m=+142.159793658" watchObservedRunningTime="2026-09-05 02:20:10.682770397 +0000 UTC m=+142.162164554"
Sep 05 02:20:10.698239 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:10.698178    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-52z87" podStartSLOduration=1.657495474 podStartE2EDuration="18.698157925s" totalImagesPullingTime="17.040662451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:10.697666712 +0000 UTC m=+142.177060873" watchObservedRunningTime="2026-09-05 02:20:10.698157925 +0000 UTC m=+142.177552117"
Sep 05 02:20:11.680468 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:11.680433    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-52z87"
Sep 05 02:20:16.492176 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:16.492136    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:16.593224 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:16.593187    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:18.869597 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:18.869558    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-747844c66d-vdz6g"]
Sep 05 02:20:19.008463 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:19.008424    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:19.110382 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:19.110330    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:19.292961 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:19.292866    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:19.359192 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:19.359158    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:19.862727 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:19.862700    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:19.888866 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:19.888823    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:20:19.964272 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:19.964230    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:26.719687 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:26.719653    2566 generic.go:356] "Generic (PLEG): container finished" podID="b2438b69-c255-4a19-98d7-56ab067c6a22" containerID="b2103636498af8d6633988618f015652475025aebfb11b9f5e4f5d3d156d7c2b" exitCode=0
Sep 05 02:20:26.720104 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:26.719712    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4r72s" event={"ID":"b2438b69-c255-4a19-98d7-56ab067c6a22","Type":"ContainerDied","Data":"b2103636498af8d6633988618f015652475025aebfb11b9f5e4f5d3d156d7c2b"}
Sep 05 02:20:26.720104 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:26.720052    2566 scope.go:117] "RemoveContainer" containerID="b2103636498af8d6633988618f015652475025aebfb11b9f5e4f5d3d156d7c2b"
Sep 05 02:20:27.725627 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:27.725592    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4r72s" event={"ID":"b2438b69-c255-4a19-98d7-56ab067c6a22","Type":"ContainerStarted","Data":"b3d4ee52c8d2caade79f96e07fa848ee14d6736f6f594872bce6d8f65cb50ed4"}
Sep 05 02:20:32.019262 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:32.019223    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:32.091677 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:32.091616    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:43.892029 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:43.891944    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-747844c66d-vdz6g" podUID="196203dc-fbce-4fc0-8835-19690ca7186d" containerName="console" containerID="cri-o://01e89b995b125d38ed172ad1e3d95ae9462a347cdde1c33d2b991e1647ca3bc4" gracePeriod=15
Sep 05 02:20:44.158757 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.158679    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:20:44.256888 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.256816    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/196203dc-fbce-4fc0-8835-19690ca7186d-oauth-serving-cert\") pod \"196203dc-fbce-4fc0-8835-19690ca7186d\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") "
Sep 05 02:20:44.256888 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.256889    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzvrp\" (UniqueName: \"kubernetes.io/projected/196203dc-fbce-4fc0-8835-19690ca7186d-kube-api-access-wzvrp\") pod \"196203dc-fbce-4fc0-8835-19690ca7186d\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") "
Sep 05 02:20:44.257141 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.256929    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/196203dc-fbce-4fc0-8835-19690ca7186d-console-serving-cert\") pod \"196203dc-fbce-4fc0-8835-19690ca7186d\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") "
Sep 05 02:20:44.257141 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.256956    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/196203dc-fbce-4fc0-8835-19690ca7186d-service-ca\") pod \"196203dc-fbce-4fc0-8835-19690ca7186d\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") "
Sep 05 02:20:44.257141 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.256984    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/196203dc-fbce-4fc0-8835-19690ca7186d-console-config\") pod \"196203dc-fbce-4fc0-8835-19690ca7186d\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") "
Sep 05 02:20:44.257141 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.257019    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/196203dc-fbce-4fc0-8835-19690ca7186d-console-oauth-config\") pod \"196203dc-fbce-4fc0-8835-19690ca7186d\" (UID: \"196203dc-fbce-4fc0-8835-19690ca7186d\") "
Sep 05 02:20:44.257383 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.257334    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/196203dc-fbce-4fc0-8835-19690ca7186d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "196203dc-fbce-4fc0-8835-19690ca7186d" (UID: "196203dc-fbce-4fc0-8835-19690ca7186d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:44.257448 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.257365    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/196203dc-fbce-4fc0-8835-19690ca7186d-service-ca" (OuterVolumeSpecName: "service-ca") pod "196203dc-fbce-4fc0-8835-19690ca7186d" (UID: "196203dc-fbce-4fc0-8835-19690ca7186d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:44.257448 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.257405    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/196203dc-fbce-4fc0-8835-19690ca7186d-console-config" (OuterVolumeSpecName: "console-config") pod "196203dc-fbce-4fc0-8835-19690ca7186d" (UID: "196203dc-fbce-4fc0-8835-19690ca7186d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:44.259271 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.259244    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196203dc-fbce-4fc0-8835-19690ca7186d-kube-api-access-wzvrp" (OuterVolumeSpecName: "kube-api-access-wzvrp") pod "196203dc-fbce-4fc0-8835-19690ca7186d" (UID: "196203dc-fbce-4fc0-8835-19690ca7186d"). InnerVolumeSpecName "kube-api-access-wzvrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:20:44.259381 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.259265    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/196203dc-fbce-4fc0-8835-19690ca7186d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "196203dc-fbce-4fc0-8835-19690ca7186d" (UID: "196203dc-fbce-4fc0-8835-19690ca7186d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:44.259381 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.259282    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/196203dc-fbce-4fc0-8835-19690ca7186d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "196203dc-fbce-4fc0-8835-19690ca7186d" (UID: "196203dc-fbce-4fc0-8835-19690ca7186d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:44.358117 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.358076    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/196203dc-fbce-4fc0-8835-19690ca7186d-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:20:44.358117 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.358107    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/196203dc-fbce-4fc0-8835-19690ca7186d-console-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:20:44.358117 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.358116    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/196203dc-fbce-4fc0-8835-19690ca7186d-console-oauth-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:20:44.358117 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.358127    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/196203dc-fbce-4fc0-8835-19690ca7186d-oauth-serving-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:20:44.358372 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.358135    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzvrp\" (UniqueName: \"kubernetes.io/projected/196203dc-fbce-4fc0-8835-19690ca7186d-kube-api-access-wzvrp\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:20:44.358372 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.358145    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/196203dc-fbce-4fc0-8835-19690ca7186d-console-serving-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:20:44.784899 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.784859    2566 generic.go:356] "Generic (PLEG): container finished" podID="196203dc-fbce-4fc0-8835-19690ca7186d" containerID="01e89b995b125d38ed172ad1e3d95ae9462a347cdde1c33d2b991e1647ca3bc4" exitCode=0
Sep 05 02:20:44.785032 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.784918    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747844c66d-vdz6g"
Sep 05 02:20:44.785032 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.784933    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747844c66d-vdz6g" event={"ID":"196203dc-fbce-4fc0-8835-19690ca7186d","Type":"ContainerDied","Data":"01e89b995b125d38ed172ad1e3d95ae9462a347cdde1c33d2b991e1647ca3bc4"}
Sep 05 02:20:44.785032 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.785005    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747844c66d-vdz6g" event={"ID":"196203dc-fbce-4fc0-8835-19690ca7186d","Type":"ContainerDied","Data":"d91fe946a16e3b4db890de66389e6ec5e513d7d349b51117f1c209049e8772d7"}
Sep 05 02:20:44.785032 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.785029    2566 scope.go:117] "RemoveContainer" containerID="01e89b995b125d38ed172ad1e3d95ae9462a347cdde1c33d2b991e1647ca3bc4"
Sep 05 02:20:44.793482 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.793463    2566 scope.go:117] "RemoveContainer" containerID="01e89b995b125d38ed172ad1e3d95ae9462a347cdde1c33d2b991e1647ca3bc4"
Sep 05 02:20:44.793731 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:20:44.793711    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01e89b995b125d38ed172ad1e3d95ae9462a347cdde1c33d2b991e1647ca3bc4\": container with ID starting with 01e89b995b125d38ed172ad1e3d95ae9462a347cdde1c33d2b991e1647ca3bc4 not found: ID does not exist" containerID="01e89b995b125d38ed172ad1e3d95ae9462a347cdde1c33d2b991e1647ca3bc4"
Sep 05 02:20:44.793781 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.793740    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01e89b995b125d38ed172ad1e3d95ae9462a347cdde1c33d2b991e1647ca3bc4"} err="failed to get container status \"01e89b995b125d38ed172ad1e3d95ae9462a347cdde1c33d2b991e1647ca3bc4\": rpc error: code = NotFound desc = could not find container \"01e89b995b125d38ed172ad1e3d95ae9462a347cdde1c33d2b991e1647ca3bc4\": container with ID starting with 01e89b995b125d38ed172ad1e3d95ae9462a347cdde1c33d2b991e1647ca3bc4 not found: ID does not exist"
Sep 05 02:20:44.804680 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.804641    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-747844c66d-vdz6g"]
Sep 05 02:20:44.808118 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:44.808095    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-747844c66d-vdz6g"]
Sep 05 02:20:45.046663 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:45.046591    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="196203dc-fbce-4fc0-8835-19690ca7186d" path="/var/lib/kubelet/pods/196203dc-fbce-4fc0-8835-19690ca7186d/volumes"
Sep 05 02:20:51.226581 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:51.226538    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:51.439521 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:51.439485    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:59.183537 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:59.183488    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:59.278797 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:20:59.278766    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:10.184924 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:21:10.184879    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:10.251891 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:21:10.251851    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:10.320891 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:21:10.320857    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:01.157534 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:22:01.157497    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:06.430215 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:22:06.430184    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:06.494782 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:22:06.494737    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:18.425330 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:22:18.425245    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:18.485675 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:22:18.485647    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:31.995362 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:22:31.995326    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:32.094654 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:22:32.094624    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:35.779209 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:22:35.779162    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:48.919169 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:22:48.919139    2566 kubelet.go:1648] "Image garbage collection succeeded"
Sep 05 02:22:53.052155 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:22:53.052120    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:53.111554 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:22:53.111514    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:03.615989 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:23:03.615957    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:03.671981 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:23:03.671955    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:03.727990 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:23:03.727961    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:15.398234 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:23:15.398196    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:15.459815 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:23:15.459785    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:04.378635 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:04.378547    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:05.911581 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:05.911535    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:26.336082 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:26.336048    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:26.606351 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:26.606263    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:26.704017 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:26.703983    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:28.343178 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:28.343146    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2q99"]
Sep 05 02:24:28.343539 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:28.343381    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="196203dc-fbce-4fc0-8835-19690ca7186d" containerName="console"
Sep 05 02:24:28.343539 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:28.343391    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="196203dc-fbce-4fc0-8835-19690ca7186d" containerName="console"
Sep 05 02:24:28.343539 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:28.343438    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="196203dc-fbce-4fc0-8835-19690ca7186d" containerName="console"
Sep 05 02:24:28.345177 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:28.345156    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2q99"
Sep 05 02:24:28.349450 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:28.349421    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 05 02:24:28.349577 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:28.349469    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 05 02:24:28.349577 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:28.349517    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 05 02:24:28.349577 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:28.349475    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-crwfh\""
Sep 05 02:24:28.356506 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:28.356483    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2q99"]
Sep 05 02:24:28.443292 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:28.443251    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ece03ab3-3e81-4269-9b8c-35fe2b7f355a-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-v2q99\" (UID: \"ece03ab3-3e81-4269-9b8c-35fe2b7f355a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2q99"
Sep 05 02:24:28.443292 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:28.443297    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6k87\" (UniqueName: \"kubernetes.io/projected/ece03ab3-3e81-4269-9b8c-35fe2b7f355a-kube-api-access-s6k87\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-v2q99\" (UID: \"ece03ab3-3e81-4269-9b8c-35fe2b7f355a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2q99"
Sep 05 02:24:28.544557 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:28.544518    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ece03ab3-3e81-4269-9b8c-35fe2b7f355a-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-v2q99\" (UID: \"ece03ab3-3e81-4269-9b8c-35fe2b7f355a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2q99"
Sep 05 02:24:28.544726 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:28.544565    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6k87\" (UniqueName: \"kubernetes.io/projected/ece03ab3-3e81-4269-9b8c-35fe2b7f355a-kube-api-access-s6k87\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-v2q99\" (UID: \"ece03ab3-3e81-4269-9b8c-35fe2b7f355a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2q99"
Sep 05 02:24:28.546849 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:28.546803    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ece03ab3-3e81-4269-9b8c-35fe2b7f355a-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-v2q99\" (UID: \"ece03ab3-3e81-4269-9b8c-35fe2b7f355a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2q99"
Sep 05 02:24:28.553607 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:28.553582    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6k87\" (UniqueName: \"kubernetes.io/projected/ece03ab3-3e81-4269-9b8c-35fe2b7f355a-kube-api-access-s6k87\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-v2q99\" (UID: \"ece03ab3-3e81-4269-9b8c-35fe2b7f355a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2q99"
Sep 05 02:24:28.654724 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:28.654685    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2q99"
Sep 05 02:24:28.927546 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:28.927469    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2q99"]
Sep 05 02:24:28.930306 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:24:28.930263    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece03ab3_3e81_4269_9b8c_35fe2b7f355a.slice/crio-f4ec203cb0858400d0253536b8761a7f0126ad14af7859c265e0a91b66986182 WatchSource:0}: Error finding container f4ec203cb0858400d0253536b8761a7f0126ad14af7859c265e0a91b66986182: Status 404 returned error can't find the container with id f4ec203cb0858400d0253536b8761a7f0126ad14af7859c265e0a91b66986182
Sep 05 02:24:28.932071 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:28.932052    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:24:29.499602 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:29.499571    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2q99" event={"ID":"ece03ab3-3e81-4269-9b8c-35fe2b7f355a","Type":"ContainerStarted","Data":"f4ec203cb0858400d0253536b8761a7f0126ad14af7859c265e0a91b66986182"}
Sep 05 02:24:32.513713 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:32.513674    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2q99" event={"ID":"ece03ab3-3e81-4269-9b8c-35fe2b7f355a","Type":"ContainerStarted","Data":"864c24f81b0aeaf72856bef8d10a35228c55aaf3159f2c0195fdd7ddfda0822d"}
Sep 05 02:24:32.514119 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:32.513823    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2q99"
Sep 05 02:24:32.535634 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:32.535577    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2q99" podStartSLOduration=1.178092255 podStartE2EDuration="4.535558111s" totalImagesPullingTime="3.357465856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:24:32.533479741 +0000 UTC m=+404.012873900" watchObservedRunningTime="2026-09-05 02:24:32.535558111 +0000 UTC m=+404.014952269"
Sep 05 02:24:32.729963 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:32.729927    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:33.065363 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:33.065272    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:33.317154 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:33.317060    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:33.484426 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:33.484397    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:33.649641 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:33.649611    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:33.717433 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:33.717388    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:33.818084 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:33.818051    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:34.048100 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:34.048015    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:34.250006 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:34.249965    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:34.406093 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:34.406058    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:34.497613 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:34.497584    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:38.831090 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:38.831039    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:53.521049 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:24:53.521012    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2q99"
Sep 05 02:25:25.770585 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:25:25.770506    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:25.870182 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:25:25.870150    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:53.756817 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:25:53.756773    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:53.884317 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:25:53.884276    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:03.402751 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.402703    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:03.475773 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.475736    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-rttd7"]
Sep 05 02:26:03.478090 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.478069    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-577b5fb6f8-rttd7"
Sep 05 02:26:03.480426 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.480399    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 05 02:26:03.480538 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.480448    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 05 02:26:03.480538 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.480484    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 05 02:26:03.481123 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.481109    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-d6vjk\""
Sep 05 02:26:03.489006 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.488979    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-mtxv7"]
Sep 05 02:26:03.490884 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.490865    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-rttd7"]
Sep 05 02:26:03.491001 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.490954    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mtxv7"
Sep 05 02:26:03.493028 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.493009    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 05 02:26:03.493138 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.493079    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-9q44q\""
Sep 05 02:26:03.498488 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.498466    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:03.501951 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.501927    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-mtxv7"]
Sep 05 02:26:03.600324 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.600291    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:03.604474 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.604450    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57e31eb9-1389-44ec-8a2b-a6c35c60829d-cert\") pod \"kserve-controller-manager-577b5fb6f8-rttd7\" (UID: \"57e31eb9-1389-44ec-8a2b-a6c35c60829d\") " pod="kserve/kserve-controller-manager-577b5fb6f8-rttd7"
Sep 05 02:26:03.604566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.604513    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb95h\" (UniqueName: \"kubernetes.io/projected/57e31eb9-1389-44ec-8a2b-a6c35c60829d-kube-api-access-zb95h\") pod \"kserve-controller-manager-577b5fb6f8-rttd7\" (UID: \"57e31eb9-1389-44ec-8a2b-a6c35c60829d\") " pod="kserve/kserve-controller-manager-577b5fb6f8-rttd7"
Sep 05 02:26:03.604566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.604542    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a1b84cf-6d08-4fc5-8356-c6a19338686f-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-mtxv7\" (UID: \"0a1b84cf-6d08-4fc5-8356-c6a19338686f\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-mtxv7"
Sep 05 02:26:03.604667 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.604573    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2478\" (UniqueName: \"kubernetes.io/projected/0a1b84cf-6d08-4fc5-8356-c6a19338686f-kube-api-access-m2478\") pod \"llmisvc-controller-manager-645f9c8fd6-mtxv7\" (UID: \"0a1b84cf-6d08-4fc5-8356-c6a19338686f\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-mtxv7"
Sep 05 02:26:03.704997 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.704908    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2478\" (UniqueName: \"kubernetes.io/projected/0a1b84cf-6d08-4fc5-8356-c6a19338686f-kube-api-access-m2478\") pod \"llmisvc-controller-manager-645f9c8fd6-mtxv7\" (UID: \"0a1b84cf-6d08-4fc5-8356-c6a19338686f\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-mtxv7"
Sep 05 02:26:03.704997 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.704972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57e31eb9-1389-44ec-8a2b-a6c35c60829d-cert\") pod \"kserve-controller-manager-577b5fb6f8-rttd7\" (UID: \"57e31eb9-1389-44ec-8a2b-a6c35c60829d\") " pod="kserve/kserve-controller-manager-577b5fb6f8-rttd7"
Sep 05 02:26:03.705212 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.705019    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb95h\" (UniqueName: \"kubernetes.io/projected/57e31eb9-1389-44ec-8a2b-a6c35c60829d-kube-api-access-zb95h\") pod \"kserve-controller-manager-577b5fb6f8-rttd7\" (UID: \"57e31eb9-1389-44ec-8a2b-a6c35c60829d\") " pod="kserve/kserve-controller-manager-577b5fb6f8-rttd7"
Sep 05 02:26:03.705212 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.705042    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a1b84cf-6d08-4fc5-8356-c6a19338686f-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-mtxv7\" (UID: \"0a1b84cf-6d08-4fc5-8356-c6a19338686f\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-mtxv7"
Sep 05 02:26:03.706227 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.706201    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:03.707642 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.707618    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57e31eb9-1389-44ec-8a2b-a6c35c60829d-cert\") pod \"kserve-controller-manager-577b5fb6f8-rttd7\" (UID: \"57e31eb9-1389-44ec-8a2b-a6c35c60829d\") " pod="kserve/kserve-controller-manager-577b5fb6f8-rttd7"
Sep 05 02:26:03.707726 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.707645    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a1b84cf-6d08-4fc5-8356-c6a19338686f-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-mtxv7\" (UID: \"0a1b84cf-6d08-4fc5-8356-c6a19338686f\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-mtxv7"
Sep 05 02:26:03.716884 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.716864    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2478\" (UniqueName: \"kubernetes.io/projected/0a1b84cf-6d08-4fc5-8356-c6a19338686f-kube-api-access-m2478\") pod \"llmisvc-controller-manager-645f9c8fd6-mtxv7\" (UID: \"0a1b84cf-6d08-4fc5-8356-c6a19338686f\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-mtxv7"
Sep 05 02:26:03.717725 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.717703    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb95h\" (UniqueName: \"kubernetes.io/projected/57e31eb9-1389-44ec-8a2b-a6c35c60829d-kube-api-access-zb95h\") pod \"kserve-controller-manager-577b5fb6f8-rttd7\" (UID: \"57e31eb9-1389-44ec-8a2b-a6c35c60829d\") " pod="kserve/kserve-controller-manager-577b5fb6f8-rttd7"
Sep 05 02:26:03.789275 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.789252    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-577b5fb6f8-rttd7"
Sep 05 02:26:03.802595 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.802573    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mtxv7"
Sep 05 02:26:03.818040 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:03.818016    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:04.064305 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:04.064238    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-rttd7"]
Sep 05 02:26:04.067198 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:26:04.067172    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e31eb9_1389_44ec_8a2b_a6c35c60829d.slice/crio-13f52bab9b9f052b0a2823d6375691326fac5578d209238486e06377f4c24373 WatchSource:0}: Error finding container 13f52bab9b9f052b0a2823d6375691326fac5578d209238486e06377f4c24373: Status 404 returned error can't find the container with id 13f52bab9b9f052b0a2823d6375691326fac5578d209238486e06377f4c24373
Sep 05 02:26:04.078295 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:04.078270    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-mtxv7"]
Sep 05 02:26:04.080971 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:26:04.080947    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0a1b84cf_6d08_4fc5_8356_c6a19338686f.slice/crio-cd0c4a09feb961c50b8e33a282f315b64eeefd64cbc119c601397eb9dfc9bd72 WatchSource:0}: Error finding container cd0c4a09feb961c50b8e33a282f315b64eeefd64cbc119c601397eb9dfc9bd72: Status 404 returned error can't find the container with id cd0c4a09feb961c50b8e33a282f315b64eeefd64cbc119c601397eb9dfc9bd72
Sep 05 02:26:04.167539 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:04.167510    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:04.259572 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:04.259539    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:04.417548 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:04.417516    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:04.517257 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:04.517227    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:04.572125 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:04.572098    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:04.829651 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:04.829524    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mtxv7" event={"ID":"0a1b84cf-6d08-4fc5-8356-c6a19338686f","Type":"ContainerStarted","Data":"cd0c4a09feb961c50b8e33a282f315b64eeefd64cbc119c601397eb9dfc9bd72"}
Sep 05 02:26:04.831064 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:04.831029    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-577b5fb6f8-rttd7" event={"ID":"57e31eb9-1389-44ec-8a2b-a6c35c60829d","Type":"ContainerStarted","Data":"13f52bab9b9f052b0a2823d6375691326fac5578d209238486e06377f4c24373"}
Sep 05 02:26:07.844203 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:07.844162    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mtxv7" event={"ID":"0a1b84cf-6d08-4fc5-8356-c6a19338686f","Type":"ContainerStarted","Data":"a68b42d42338e64916c6bcab774810886671c96cf7fc94b56509fd7f143a5e6c"}
Sep 05 02:26:07.844623 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:07.844282    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mtxv7"
Sep 05 02:26:07.845617 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:07.845595    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-577b5fb6f8-rttd7" event={"ID":"57e31eb9-1389-44ec-8a2b-a6c35c60829d","Type":"ContainerStarted","Data":"28009be783f41003b4897d68f3f1c9ad4d65ccd18b9f1c2c827a39825d7d59b4"}
Sep 05 02:26:07.845729 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:07.845717    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-577b5fb6f8-rttd7"
Sep 05 02:26:07.860489 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:07.860448    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mtxv7" podStartSLOduration=2.118361413 podStartE2EDuration="4.860434958s" totalImagesPullingTime="2.742073545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:26:07.859479487 +0000 UTC m=+499.338873659" watchObservedRunningTime="2026-09-05 02:26:07.860434958 +0000 UTC m=+499.339829113"
Sep 05 02:26:07.876619 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:07.876577    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-577b5fb6f8-rttd7" podStartSLOduration=2.122750726 podStartE2EDuration="4.876564668s" totalImagesPullingTime="2.753813942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:26:07.874986124 +0000 UTC m=+499.354380282" watchObservedRunningTime="2026-09-05 02:26:07.876564668 +0000 UTC m=+499.355958826"
Sep 05 02:26:38.851192 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:38.851160    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mtxv7"
Sep 05 02:26:38.854181 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:38.854158    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-577b5fb6f8-rttd7"
Sep 05 02:26:46.641611 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:46.641499    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:47.004462 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.004378    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-rttd7"]
Sep 05 02:26:47.004606 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.004578    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-577b5fb6f8-rttd7" podUID="57e31eb9-1389-44ec-8a2b-a6c35c60829d" containerName="manager" containerID="cri-o://28009be783f41003b4897d68f3f1c9ad4d65ccd18b9f1c2c827a39825d7d59b4" gracePeriod=10
Sep 05 02:26:47.078468 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.078441    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-x6g4d"]
Sep 05 02:26:47.084424 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.084396    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-577b5fb6f8-x6g4d"
Sep 05 02:26:47.095925 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.095896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5d218bf-9dd4-40f8-a91c-b61bbbb65782-cert\") pod \"kserve-controller-manager-577b5fb6f8-x6g4d\" (UID: \"d5d218bf-9dd4-40f8-a91c-b61bbbb65782\") " pod="kserve/kserve-controller-manager-577b5fb6f8-x6g4d"
Sep 05 02:26:47.096036 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.095954    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2szcj\" (UniqueName: \"kubernetes.io/projected/d5d218bf-9dd4-40f8-a91c-b61bbbb65782-kube-api-access-2szcj\") pod \"kserve-controller-manager-577b5fb6f8-x6g4d\" (UID: \"d5d218bf-9dd4-40f8-a91c-b61bbbb65782\") " pod="kserve/kserve-controller-manager-577b5fb6f8-x6g4d"
Sep 05 02:26:47.099305 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.099280    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-x6g4d"]
Sep 05 02:26:47.196363 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.196325    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2szcj\" (UniqueName: \"kubernetes.io/projected/d5d218bf-9dd4-40f8-a91c-b61bbbb65782-kube-api-access-2szcj\") pod \"kserve-controller-manager-577b5fb6f8-x6g4d\" (UID: \"d5d218bf-9dd4-40f8-a91c-b61bbbb65782\") " pod="kserve/kserve-controller-manager-577b5fb6f8-x6g4d"
Sep 05 02:26:47.196499 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.196403    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5d218bf-9dd4-40f8-a91c-b61bbbb65782-cert\") pod \"kserve-controller-manager-577b5fb6f8-x6g4d\" (UID: \"d5d218bf-9dd4-40f8-a91c-b61bbbb65782\") " pod="kserve/kserve-controller-manager-577b5fb6f8-x6g4d"
Sep 05 02:26:47.198733 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.198704    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5d218bf-9dd4-40f8-a91c-b61bbbb65782-cert\") pod \"kserve-controller-manager-577b5fb6f8-x6g4d\" (UID: \"d5d218bf-9dd4-40f8-a91c-b61bbbb65782\") " pod="kserve/kserve-controller-manager-577b5fb6f8-x6g4d"
Sep 05 02:26:47.220128 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.220098    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2szcj\" (UniqueName: \"kubernetes.io/projected/d5d218bf-9dd4-40f8-a91c-b61bbbb65782-kube-api-access-2szcj\") pod \"kserve-controller-manager-577b5fb6f8-x6g4d\" (UID: \"d5d218bf-9dd4-40f8-a91c-b61bbbb65782\") " pod="kserve/kserve-controller-manager-577b5fb6f8-x6g4d"
Sep 05 02:26:47.245975 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.245952    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-577b5fb6f8-rttd7"
Sep 05 02:26:47.296680 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.296608    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57e31eb9-1389-44ec-8a2b-a6c35c60829d-cert\") pod \"57e31eb9-1389-44ec-8a2b-a6c35c60829d\" (UID: \"57e31eb9-1389-44ec-8a2b-a6c35c60829d\") "
Sep 05 02:26:47.296680 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.296641    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb95h\" (UniqueName: \"kubernetes.io/projected/57e31eb9-1389-44ec-8a2b-a6c35c60829d-kube-api-access-zb95h\") pod \"57e31eb9-1389-44ec-8a2b-a6c35c60829d\" (UID: \"57e31eb9-1389-44ec-8a2b-a6c35c60829d\") "
Sep 05 02:26:47.298786 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.298753    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57e31eb9-1389-44ec-8a2b-a6c35c60829d-cert" (OuterVolumeSpecName: "cert") pod "57e31eb9-1389-44ec-8a2b-a6c35c60829d" (UID: "57e31eb9-1389-44ec-8a2b-a6c35c60829d"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:26:47.298912 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.298808    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e31eb9-1389-44ec-8a2b-a6c35c60829d-kube-api-access-zb95h" (OuterVolumeSpecName: "kube-api-access-zb95h") pod "57e31eb9-1389-44ec-8a2b-a6c35c60829d" (UID: "57e31eb9-1389-44ec-8a2b-a6c35c60829d"). InnerVolumeSpecName "kube-api-access-zb95h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:26:47.393531 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.393493    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-577b5fb6f8-x6g4d"
Sep 05 02:26:47.397516 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.397491    2566 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57e31eb9-1389-44ec-8a2b-a6c35c60829d-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:26:47.397570 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.397525    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb95h\" (UniqueName: \"kubernetes.io/projected/57e31eb9-1389-44ec-8a2b-a6c35c60829d-kube-api-access-zb95h\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:26:47.664997 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.664970    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-x6g4d"]
Sep 05 02:26:47.667204 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:26:47.667177    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d218bf_9dd4_40f8_a91c_b61bbbb65782.slice/crio-aefff42657ca6dda1345fd777de99ea3d62704379406ccd32820220236cf16b4 WatchSource:0}: Error finding container aefff42657ca6dda1345fd777de99ea3d62704379406ccd32820220236cf16b4: Status 404 returned error can't find the container with id aefff42657ca6dda1345fd777de99ea3d62704379406ccd32820220236cf16b4
Sep 05 02:26:47.983372 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.983290    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-577b5fb6f8-x6g4d" event={"ID":"d5d218bf-9dd4-40f8-a91c-b61bbbb65782","Type":"ContainerStarted","Data":"aefff42657ca6dda1345fd777de99ea3d62704379406ccd32820220236cf16b4"}
Sep 05 02:26:47.984433 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.984408    2566 generic.go:356] "Generic (PLEG): container finished" podID="57e31eb9-1389-44ec-8a2b-a6c35c60829d" containerID="28009be783f41003b4897d68f3f1c9ad4d65ccd18b9f1c2c827a39825d7d59b4" exitCode=0
Sep 05 02:26:47.984573 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.984444    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-577b5fb6f8-rttd7" event={"ID":"57e31eb9-1389-44ec-8a2b-a6c35c60829d","Type":"ContainerDied","Data":"28009be783f41003b4897d68f3f1c9ad4d65ccd18b9f1c2c827a39825d7d59b4"}
Sep 05 02:26:47.984573 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.984454    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-577b5fb6f8-rttd7"
Sep 05 02:26:47.984573 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.984481    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-577b5fb6f8-rttd7" event={"ID":"57e31eb9-1389-44ec-8a2b-a6c35c60829d","Type":"ContainerDied","Data":"13f52bab9b9f052b0a2823d6375691326fac5578d209238486e06377f4c24373"}
Sep 05 02:26:47.984573 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.984510    2566 scope.go:117] "RemoveContainer" containerID="28009be783f41003b4897d68f3f1c9ad4d65ccd18b9f1c2c827a39825d7d59b4"
Sep 05 02:26:47.997791 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.997769    2566 scope.go:117] "RemoveContainer" containerID="28009be783f41003b4897d68f3f1c9ad4d65ccd18b9f1c2c827a39825d7d59b4"
Sep 05 02:26:47.998075 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:26:47.998051    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28009be783f41003b4897d68f3f1c9ad4d65ccd18b9f1c2c827a39825d7d59b4\": container with ID starting with 28009be783f41003b4897d68f3f1c9ad4d65ccd18b9f1c2c827a39825d7d59b4 not found: ID does not exist" containerID="28009be783f41003b4897d68f3f1c9ad4d65ccd18b9f1c2c827a39825d7d59b4"
Sep 05 02:26:47.998166 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:47.998080    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28009be783f41003b4897d68f3f1c9ad4d65ccd18b9f1c2c827a39825d7d59b4"} err="failed to get container status \"28009be783f41003b4897d68f3f1c9ad4d65ccd18b9f1c2c827a39825d7d59b4\": rpc error: code = NotFound desc = could not find container \"28009be783f41003b4897d68f3f1c9ad4d65ccd18b9f1c2c827a39825d7d59b4\": container with ID starting with 28009be783f41003b4897d68f3f1c9ad4d65ccd18b9f1c2c827a39825d7d59b4 not found: ID does not exist"
Sep 05 02:26:48.005394 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:48.005375    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-rttd7"]
Sep 05 02:26:48.008331 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:48.008310    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-rttd7"]
Sep 05 02:26:48.988994 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:48.988961    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-577b5fb6f8-x6g4d" event={"ID":"d5d218bf-9dd4-40f8-a91c-b61bbbb65782","Type":"ContainerStarted","Data":"0e3f9970452f2e70996a0f2a56a6243a280a985b27e52b1e8fb88545e0d510ab"}
Sep 05 02:26:48.989305 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:48.989023    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-577b5fb6f8-x6g4d"
Sep 05 02:26:49.007352 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:49.007306    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-577b5fb6f8-x6g4d" podStartSLOduration=1.674670473 podStartE2EDuration="2.007290315s" totalImagesPullingTime="332.619842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:26:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:26:49.006456096 +0000 UTC m=+540.485850255" watchObservedRunningTime="2026-09-05 02:26:49.007290315 +0000 UTC m=+540.486684473"
Sep 05 02:26:49.046844 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:26:49.046799    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57e31eb9-1389-44ec-8a2b-a6c35c60829d" path="/var/lib/kubelet/pods/57e31eb9-1389-44ec-8a2b-a6c35c60829d/volumes"
Sep 05 02:27:13.047858 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:27:13.047815    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:19.997708 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:27:19.997675    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-577b5fb6f8-x6g4d"
Sep 05 02:27:21.272470 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:27:21.272423    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:21.375283 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:27:21.375252    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:21.475278 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:27:21.475225    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:21.577056 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:27:21.576983    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:21.698325 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:27:21.698285    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:22.641883 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:27:22.641847    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:22.698619 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:27:22.698582    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:56.425474 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:28:56.425430    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:56.550594 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:28:56.550561    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:29:10.044110 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:10.044074    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:29:12.411418 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:12.411381    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:29:38.402933 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.402896    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:29:38.529140 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.529109    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:29:38.684568 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.684483    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:29:38.778501 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.778460    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:29:38.782940 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.782916    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"]
Sep 05 02:29:38.783244 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.783228    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e31eb9-1389-44ec-8a2b-a6c35c60829d" containerName="manager"
Sep 05 02:29:38.783313 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.783248    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e31eb9-1389-44ec-8a2b-a6c35c60829d" containerName="manager"
Sep 05 02:29:38.783367 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.783319    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e31eb9-1389-44ec-8a2b-a6c35c60829d" containerName="manager"
Sep 05 02:29:38.786272 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.786254    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"
Sep 05 02:29:38.788475 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.788450    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a6d98-predictor-serving-cert\""
Sep 05 02:29:38.788585 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.788500    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a6d98-kube-rbac-proxy-sar-config\""
Sep 05 02:29:38.788998 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.788983    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7f65m\""
Sep 05 02:29:38.789372 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.789357    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:29:38.789453 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.789426    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:29:38.794596 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.794574    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"]
Sep 05 02:29:38.862414 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.862368    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:29:38.867371 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.867343    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpgs5\" (UniqueName: \"kubernetes.io/projected/f5653e59-452f-470e-b514-988ffc7c106c-kube-api-access-hpgs5\") pod \"error-404-isvc-a6d98-predictor-697b88f8b7-9wscs\" (UID: \"f5653e59-452f-470e-b514-988ffc7c106c\") " pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"
Sep 05 02:29:38.867503 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.867413    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5653e59-452f-470e-b514-988ffc7c106c-proxy-tls\") pod \"error-404-isvc-a6d98-predictor-697b88f8b7-9wscs\" (UID: \"f5653e59-452f-470e-b514-988ffc7c106c\") " pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"
Sep 05 02:29:38.867503 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.867445    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5653e59-452f-470e-b514-988ffc7c106c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a6d98-predictor-697b88f8b7-9wscs\" (UID: \"f5653e59-452f-470e-b514-988ffc7c106c\") " pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"
Sep 05 02:29:38.968264 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.968190    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5653e59-452f-470e-b514-988ffc7c106c-proxy-tls\") pod \"error-404-isvc-a6d98-predictor-697b88f8b7-9wscs\" (UID: \"f5653e59-452f-470e-b514-988ffc7c106c\") " pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"
Sep 05 02:29:38.968264 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.968228    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5653e59-452f-470e-b514-988ffc7c106c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a6d98-predictor-697b88f8b7-9wscs\" (UID: \"f5653e59-452f-470e-b514-988ffc7c106c\") " pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"
Sep 05 02:29:38.968480 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.968271    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpgs5\" (UniqueName: \"kubernetes.io/projected/f5653e59-452f-470e-b514-988ffc7c106c-kube-api-access-hpgs5\") pod \"error-404-isvc-a6d98-predictor-697b88f8b7-9wscs\" (UID: \"f5653e59-452f-470e-b514-988ffc7c106c\") " pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"
Sep 05 02:29:38.968917 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.968893    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5653e59-452f-470e-b514-988ffc7c106c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a6d98-predictor-697b88f8b7-9wscs\" (UID: \"f5653e59-452f-470e-b514-988ffc7c106c\") " pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"
Sep 05 02:29:38.970697 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.970671    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5653e59-452f-470e-b514-988ffc7c106c-proxy-tls\") pod \"error-404-isvc-a6d98-predictor-697b88f8b7-9wscs\" (UID: \"f5653e59-452f-470e-b514-988ffc7c106c\") " pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"
Sep 05 02:29:38.976310 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.976288    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"]
Sep 05 02:29:38.979895 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.979876    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:29:38.981710 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.981686    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpgs5\" (UniqueName: \"kubernetes.io/projected/f5653e59-452f-470e-b514-988ffc7c106c-kube-api-access-hpgs5\") pod \"error-404-isvc-a6d98-predictor-697b88f8b7-9wscs\" (UID: \"f5653e59-452f-470e-b514-988ffc7c106c\") " pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"
Sep 05 02:29:38.982041 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.982025    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Sep 05 02:29:38.982183 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.982162    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Sep 05 02:29:38.982279 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.982262    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:29:38.982468 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.982456    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:29:38.992548 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:38.992130    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"]
Sep 05 02:29:39.027486 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.027456    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:29:39.069399 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.069375    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe25f418-9579-48f2-b9da-cd28e490c3b5-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj\" (UID: \"fe25f418-9579-48f2-b9da-cd28e490c3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:29:39.069530 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.069417    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe25f418-9579-48f2-b9da-cd28e490c3b5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj\" (UID: \"fe25f418-9579-48f2-b9da-cd28e490c3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:29:39.069530 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.069441    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe25f418-9579-48f2-b9da-cd28e490c3b5-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj\" (UID: \"fe25f418-9579-48f2-b9da-cd28e490c3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:29:39.069530 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.069512    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe25f418-9579-48f2-b9da-cd28e490c3b5-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj\" (UID: \"fe25f418-9579-48f2-b9da-cd28e490c3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:29:39.069649 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.069545    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8zfz\" (UniqueName: \"kubernetes.io/projected/fe25f418-9579-48f2-b9da-cd28e490c3b5-kube-api-access-s8zfz\") pod \"isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj\" (UID: \"fe25f418-9579-48f2-b9da-cd28e490c3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:29:39.097408 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.097374    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"
Sep 05 02:29:39.149909 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.149864    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:29:39.170296 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.170266    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe25f418-9579-48f2-b9da-cd28e490c3b5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj\" (UID: \"fe25f418-9579-48f2-b9da-cd28e490c3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:29:39.170416 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.170304    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe25f418-9579-48f2-b9da-cd28e490c3b5-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj\" (UID: \"fe25f418-9579-48f2-b9da-cd28e490c3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:29:39.170416 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.170368    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe25f418-9579-48f2-b9da-cd28e490c3b5-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj\" (UID: \"fe25f418-9579-48f2-b9da-cd28e490c3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:29:39.170416 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.170391    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8zfz\" (UniqueName: \"kubernetes.io/projected/fe25f418-9579-48f2-b9da-cd28e490c3b5-kube-api-access-s8zfz\") pod \"isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj\" (UID: \"fe25f418-9579-48f2-b9da-cd28e490c3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:29:39.170544 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.170434    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe25f418-9579-48f2-b9da-cd28e490c3b5-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj\" (UID: \"fe25f418-9579-48f2-b9da-cd28e490c3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:29:39.171068 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.171040    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe25f418-9579-48f2-b9da-cd28e490c3b5-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj\" (UID: \"fe25f418-9579-48f2-b9da-cd28e490c3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:29:39.171166 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.171092    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe25f418-9579-48f2-b9da-cd28e490c3b5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj\" (UID: \"fe25f418-9579-48f2-b9da-cd28e490c3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:29:39.171376 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.171352    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe25f418-9579-48f2-b9da-cd28e490c3b5-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj\" (UID: \"fe25f418-9579-48f2-b9da-cd28e490c3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:29:39.173864 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.173684    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe25f418-9579-48f2-b9da-cd28e490c3b5-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj\" (UID: \"fe25f418-9579-48f2-b9da-cd28e490c3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:29:39.179701 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.179675    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8zfz\" (UniqueName: \"kubernetes.io/projected/fe25f418-9579-48f2-b9da-cd28e490c3b5-kube-api-access-s8zfz\") pod \"isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj\" (UID: \"fe25f418-9579-48f2-b9da-cd28e490c3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:29:39.296774 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.296687    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:29:39.375766 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.375742    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"]
Sep 05 02:29:39.378184 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:29:39.378158    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5653e59_452f_470e_b514_988ffc7c106c.slice/crio-c2723aebaaca9b75090802c85162daf687f41f182c11ecf75619d743acf7bb51 WatchSource:0}: Error finding container c2723aebaaca9b75090802c85162daf687f41f182c11ecf75619d743acf7bb51: Status 404 returned error can't find the container with id c2723aebaaca9b75090802c85162daf687f41f182c11ecf75619d743acf7bb51
Sep 05 02:29:39.380119 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.380103    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:29:39.568681 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.568560    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"]
Sep 05 02:29:39.570544 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:39.570512    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs" event={"ID":"f5653e59-452f-470e-b514-988ffc7c106c","Type":"ContainerStarted","Data":"c2723aebaaca9b75090802c85162daf687f41f182c11ecf75619d743acf7bb51"}
Sep 05 02:29:39.571365 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:29:39.571342    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe25f418_9579_48f2_b9da_cd28e490c3b5.slice/crio-735aba264dff81b6ac191dd5c5f084185d698ba67cd1950aab6487ad79335988 WatchSource:0}: Error finding container 735aba264dff81b6ac191dd5c5f084185d698ba67cd1950aab6487ad79335988: Status 404 returned error can't find the container with id 735aba264dff81b6ac191dd5c5f084185d698ba67cd1950aab6487ad79335988
Sep 05 02:29:40.580005 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:40.579945    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" event={"ID":"fe25f418-9579-48f2-b9da-cd28e490c3b5","Type":"ContainerStarted","Data":"735aba264dff81b6ac191dd5c5f084185d698ba67cd1950aab6487ad79335988"}
Sep 05 02:29:50.634929 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:50.634895    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:29:52.639715 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:52.639676    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" event={"ID":"fe25f418-9579-48f2-b9da-cd28e490c3b5","Type":"ContainerStarted","Data":"a372c8aeef69be38962bdf52736b49fc04ef4f44529fd7407a4754ef6e81a2dc"}
Sep 05 02:29:52.641279 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:52.641255    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs" event={"ID":"f5653e59-452f-470e-b514-988ffc7c106c","Type":"ContainerStarted","Data":"54fbf9ce2c8217226e5cb050f3cac1a453f288c8200720551d76e9d309fedc0a"}
Sep 05 02:29:55.655316 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:55.655273    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs" event={"ID":"f5653e59-452f-470e-b514-988ffc7c106c","Type":"ContainerStarted","Data":"0786ab948341ff1c5ddfbfe376b773716959ef1d894a3fb5de590ceccd580674"}
Sep 05 02:29:55.655719 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:55.655368    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"
Sep 05 02:29:55.655719 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:55.655392    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"
Sep 05 02:29:55.656772 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:55.656738    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs" podUID="f5653e59-452f-470e-b514-988ffc7c106c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 05 02:29:55.674655 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:55.674577    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs" podStartSLOduration=2.451622571 podStartE2EDuration="17.674550959s" totalImagesPullingTime="15.222928388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:29:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:29:55.673245177 +0000 UTC m=+727.152639336" watchObservedRunningTime="2026-09-05 02:29:55.674550959 +0000 UTC m=+727.153945177"
Sep 05 02:29:56.660441 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:56.660408    2566 generic.go:356] "Generic (PLEG): container finished" podID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerID="a372c8aeef69be38962bdf52736b49fc04ef4f44529fd7407a4754ef6e81a2dc" exitCode=0
Sep 05 02:29:56.660805 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:56.660484    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" event={"ID":"fe25f418-9579-48f2-b9da-cd28e490c3b5","Type":"ContainerDied","Data":"a372c8aeef69be38962bdf52736b49fc04ef4f44529fd7407a4754ef6e81a2dc"}
Sep 05 02:29:56.660805 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:29:56.660765    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs" podUID="f5653e59-452f-470e-b514-988ffc7c106c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 05 02:30:01.666142 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:30:01.666104    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"
Sep 05 02:30:01.666630 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:30:01.666555    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs" podUID="f5653e59-452f-470e-b514-988ffc7c106c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 05 02:30:02.686006 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:30:02.685976    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" event={"ID":"fe25f418-9579-48f2-b9da-cd28e490c3b5","Type":"ContainerStarted","Data":"630ea1d77c830483e8e54bcf1e51bcf760794a5af1bc7aca03f894ebee14bd25"}
Sep 05 02:30:03.691635 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:30:03.691598    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" event={"ID":"fe25f418-9579-48f2-b9da-cd28e490c3b5","Type":"ContainerStarted","Data":"acacf73dff8ff16edd0d3ac2b63c029d707f7b8e2b75006a03cbc116b17a0bf4"}
Sep 05 02:30:03.692093 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:30:03.691856    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:30:03.711236 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:30:03.711187    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" podStartSLOduration=4.609814567 podStartE2EDuration="25.711174388s" totalImagesPullingTime="16.990358903s" totalInitContainerRuntime="4.111000918s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:29:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:30:03.710581657 +0000 UTC m=+735.189975817" watchObservedRunningTime="2026-09-05 02:30:03.711174388 +0000 UTC m=+735.190568545"
Sep 05 02:30:04.695179 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:30:04.695132    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:30:04.696377 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:30:04.696347    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 05 02:30:05.699020 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:30:05.698975    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 05 02:30:10.704373 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:30:10.704340    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:30:10.704943 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:30:10.704917    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 05 02:30:11.666603 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:30:11.666562    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs" podUID="f5653e59-452f-470e-b514-988ffc7c106c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 05 02:30:20.705204 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:30:20.705159    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 05 02:30:21.666936 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:30:21.666898    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs" podUID="f5653e59-452f-470e-b514-988ffc7c106c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 05 02:30:30.705844 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:30:30.705790    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 05 02:30:31.667033 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:30:31.666985    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs" podUID="f5653e59-452f-470e-b514-988ffc7c106c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 05 02:30:40.705489 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:30:40.705446    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 05 02:30:41.667560 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:30:41.667530    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"
Sep 05 02:30:50.705871 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:30:50.705817    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 05 02:30:58.495578 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:30:58.495540    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:30:58.593923 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:30:58.593887    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:00.705092 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:00.705045    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 05 02:31:08.518233 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:08.518194    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:08.720996 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:08.720964    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:08.809881 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:08.809758    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:08.810258 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:08.810234    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"]
Sep 05 02:31:08.810508 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:08.810470    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs" podUID="f5653e59-452f-470e-b514-988ffc7c106c" containerName="kserve-container" containerID="cri-o://54fbf9ce2c8217226e5cb050f3cac1a453f288c8200720551d76e9d309fedc0a" gracePeriod=30
Sep 05 02:31:08.810608 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:08.810498    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs" podUID="f5653e59-452f-470e-b514-988ffc7c106c" containerName="kube-rbac-proxy" containerID="cri-o://0786ab948341ff1c5ddfbfe376b773716959ef1d894a3fb5de590ceccd580674" gracePeriod=30
Sep 05 02:31:08.888746 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:08.888711    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:08.931375 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:08.931345    2566 generic.go:356] "Generic (PLEG): container finished" podID="f5653e59-452f-470e-b514-988ffc7c106c" containerID="0786ab948341ff1c5ddfbfe376b773716959ef1d894a3fb5de590ceccd580674" exitCode=2
Sep 05 02:31:08.931493 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:08.931417    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs" event={"ID":"f5653e59-452f-470e-b514-988ffc7c106c","Type":"ContainerDied","Data":"0786ab948341ff1c5ddfbfe376b773716959ef1d894a3fb5de590ceccd580674"}
Sep 05 02:31:08.986897 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:08.986862    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:09.035313 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:09.035285    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"]
Sep 05 02:31:09.038459 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:09.038442    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"
Sep 05 02:31:09.040802 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:09.040781    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3f635-predictor-serving-cert\""
Sep 05 02:31:09.040939 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:09.040814    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3f635-kube-rbac-proxy-sar-config\""
Sep 05 02:31:09.049698 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:09.049673    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"]
Sep 05 02:31:09.078331 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:09.078234    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:09.146532 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:09.146498    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44f9ca6b-951b-4716-b11a-2c8d478c9a95-proxy-tls\") pod \"error-404-isvc-3f635-predictor-bd67df8df-mxddc\" (UID: \"44f9ca6b-951b-4716-b11a-2c8d478c9a95\") " pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"
Sep 05 02:31:09.146687 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:09.146625    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqmpp\" (UniqueName: \"kubernetes.io/projected/44f9ca6b-951b-4716-b11a-2c8d478c9a95-kube-api-access-rqmpp\") pod \"error-404-isvc-3f635-predictor-bd67df8df-mxddc\" (UID: \"44f9ca6b-951b-4716-b11a-2c8d478c9a95\") " pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"
Sep 05 02:31:09.146736 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:09.146682    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44f9ca6b-951b-4716-b11a-2c8d478c9a95-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3f635-predictor-bd67df8df-mxddc\" (UID: \"44f9ca6b-951b-4716-b11a-2c8d478c9a95\") " pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"
Sep 05 02:31:09.247979 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:09.247936    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqmpp\" (UniqueName: \"kubernetes.io/projected/44f9ca6b-951b-4716-b11a-2c8d478c9a95-kube-api-access-rqmpp\") pod \"error-404-isvc-3f635-predictor-bd67df8df-mxddc\" (UID: \"44f9ca6b-951b-4716-b11a-2c8d478c9a95\") " pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"
Sep 05 02:31:09.248116 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:09.248000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44f9ca6b-951b-4716-b11a-2c8d478c9a95-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3f635-predictor-bd67df8df-mxddc\" (UID: \"44f9ca6b-951b-4716-b11a-2c8d478c9a95\") " pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"
Sep 05 02:31:09.248116 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:09.248031    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44f9ca6b-951b-4716-b11a-2c8d478c9a95-proxy-tls\") pod \"error-404-isvc-3f635-predictor-bd67df8df-mxddc\" (UID: \"44f9ca6b-951b-4716-b11a-2c8d478c9a95\") " pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"
Sep 05 02:31:09.248237 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:31:09.248143    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-3f635-predictor-serving-cert: secret "error-404-isvc-3f635-predictor-serving-cert" not found
Sep 05 02:31:09.248237 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:31:09.248204    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44f9ca6b-951b-4716-b11a-2c8d478c9a95-proxy-tls podName:44f9ca6b-951b-4716-b11a-2c8d478c9a95 nodeName:}" failed. No retries permitted until 2026-09-05 02:31:09.748184729 +0000 UTC m=+801.227578880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/44f9ca6b-951b-4716-b11a-2c8d478c9a95-proxy-tls") pod "error-404-isvc-3f635-predictor-bd67df8df-mxddc" (UID: "44f9ca6b-951b-4716-b11a-2c8d478c9a95") : secret "error-404-isvc-3f635-predictor-serving-cert" not found
Sep 05 02:31:09.248662 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:09.248643    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44f9ca6b-951b-4716-b11a-2c8d478c9a95-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3f635-predictor-bd67df8df-mxddc\" (UID: \"44f9ca6b-951b-4716-b11a-2c8d478c9a95\") " pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"
Sep 05 02:31:09.257424 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:09.257396    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqmpp\" (UniqueName: \"kubernetes.io/projected/44f9ca6b-951b-4716-b11a-2c8d478c9a95-kube-api-access-rqmpp\") pod \"error-404-isvc-3f635-predictor-bd67df8df-mxddc\" (UID: \"44f9ca6b-951b-4716-b11a-2c8d478c9a95\") " pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"
Sep 05 02:31:09.298306 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:09.298277    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:09.443698 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:09.443641    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:09.752384 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:09.752292    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44f9ca6b-951b-4716-b11a-2c8d478c9a95-proxy-tls\") pod \"error-404-isvc-3f635-predictor-bd67df8df-mxddc\" (UID: \"44f9ca6b-951b-4716-b11a-2c8d478c9a95\") " pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"
Sep 05 02:31:09.754637 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:09.754612    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44f9ca6b-951b-4716-b11a-2c8d478c9a95-proxy-tls\") pod \"error-404-isvc-3f635-predictor-bd67df8df-mxddc\" (UID: \"44f9ca6b-951b-4716-b11a-2c8d478c9a95\") " pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"
Sep 05 02:31:09.949247 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:09.949218    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"
Sep 05 02:31:10.217103 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:10.217074    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"]
Sep 05 02:31:10.219218 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:31:10.219185    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f9ca6b_951b_4716_b11a_2c8d478c9a95.slice/crio-4e42e9a68bbc4471bbecd0a6457450ec41b54d898b8621c057466ee61d8e9476 WatchSource:0}: Error finding container 4e42e9a68bbc4471bbecd0a6457450ec41b54d898b8621c057466ee61d8e9476: Status 404 returned error can't find the container with id 4e42e9a68bbc4471bbecd0a6457450ec41b54d898b8621c057466ee61d8e9476
Sep 05 02:31:10.705330 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:10.705291    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 05 02:31:10.940073 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:10.940040    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc" event={"ID":"44f9ca6b-951b-4716-b11a-2c8d478c9a95","Type":"ContainerStarted","Data":"bb3a0fd055c590cfc3e33f65eb8cf3db605dbd68d32882cf0b43b075a71b355b"}
Sep 05 02:31:10.940454 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:10.940100    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc" event={"ID":"44f9ca6b-951b-4716-b11a-2c8d478c9a95","Type":"ContainerStarted","Data":"12638a228971c5a8a28dfeedfac207790534dc095d0bf8a869667dd787910f09"}
Sep 05 02:31:10.940454 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:10.940132    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc" event={"ID":"44f9ca6b-951b-4716-b11a-2c8d478c9a95","Type":"ContainerStarted","Data":"4e42e9a68bbc4471bbecd0a6457450ec41b54d898b8621c057466ee61d8e9476"}
Sep 05 02:31:10.940454 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:10.940156    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"
Sep 05 02:31:10.940454 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:10.940175    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"
Sep 05 02:31:10.941423 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:10.941398    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc" podUID="44f9ca6b-951b-4716-b11a-2c8d478c9a95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 05 02:31:10.957928 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:10.957846    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc" podStartSLOduration=1.957817154 podStartE2EDuration="1.957817154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:31:10.95627708 +0000 UTC m=+802.435671239" watchObservedRunningTime="2026-09-05 02:31:10.957817154 +0000 UTC m=+802.437211311"
Sep 05 02:31:11.661731 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:11.661688    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs" podUID="f5653e59-452f-470e-b514-988ffc7c106c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Sep 05 02:31:11.666852 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:11.666804    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs" podUID="f5653e59-452f-470e-b514-988ffc7c106c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 05 02:31:11.946231 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:11.946203    2566 generic.go:356] "Generic (PLEG): container finished" podID="f5653e59-452f-470e-b514-988ffc7c106c" containerID="54fbf9ce2c8217226e5cb050f3cac1a453f288c8200720551d76e9d309fedc0a" exitCode=0
Sep 05 02:31:11.946534 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:11.946236    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs" event={"ID":"f5653e59-452f-470e-b514-988ffc7c106c","Type":"ContainerDied","Data":"54fbf9ce2c8217226e5cb050f3cac1a453f288c8200720551d76e9d309fedc0a"}
Sep 05 02:31:11.946603 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:11.946582    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc" podUID="44f9ca6b-951b-4716-b11a-2c8d478c9a95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 05 02:31:12.057450 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:12.057426    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"
Sep 05 02:31:12.172493 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:12.172409    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5653e59-452f-470e-b514-988ffc7c106c-kube-rbac-proxy-sar-config\") pod \"f5653e59-452f-470e-b514-988ffc7c106c\" (UID: \"f5653e59-452f-470e-b514-988ffc7c106c\") "
Sep 05 02:31:12.172493 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:12.172463    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5653e59-452f-470e-b514-988ffc7c106c-proxy-tls\") pod \"f5653e59-452f-470e-b514-988ffc7c106c\" (UID: \"f5653e59-452f-470e-b514-988ffc7c106c\") "
Sep 05 02:31:12.172709 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:12.172495    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpgs5\" (UniqueName: \"kubernetes.io/projected/f5653e59-452f-470e-b514-988ffc7c106c-kube-api-access-hpgs5\") pod \"f5653e59-452f-470e-b514-988ffc7c106c\" (UID: \"f5653e59-452f-470e-b514-988ffc7c106c\") "
Sep 05 02:31:12.172766 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:12.172733    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5653e59-452f-470e-b514-988ffc7c106c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f5653e59-452f-470e-b514-988ffc7c106c" (UID: "f5653e59-452f-470e-b514-988ffc7c106c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:31:12.174660 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:12.174635    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5653e59-452f-470e-b514-988ffc7c106c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f5653e59-452f-470e-b514-988ffc7c106c" (UID: "f5653e59-452f-470e-b514-988ffc7c106c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:31:12.174743 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:12.174636    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5653e59-452f-470e-b514-988ffc7c106c-kube-api-access-hpgs5" (OuterVolumeSpecName: "kube-api-access-hpgs5") pod "f5653e59-452f-470e-b514-988ffc7c106c" (UID: "f5653e59-452f-470e-b514-988ffc7c106c"). InnerVolumeSpecName "kube-api-access-hpgs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:31:12.273060 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:12.273028    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpgs5\" (UniqueName: \"kubernetes.io/projected/f5653e59-452f-470e-b514-988ffc7c106c-kube-api-access-hpgs5\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:31:12.273060 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:12.273055    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5653e59-452f-470e-b514-988ffc7c106c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:31:12.273060 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:12.273065    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5653e59-452f-470e-b514-988ffc7c106c-proxy-tls\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:31:12.951505 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:12.951475    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"
Sep 05 02:31:12.951505 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:12.951499    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs" event={"ID":"f5653e59-452f-470e-b514-988ffc7c106c","Type":"ContainerDied","Data":"c2723aebaaca9b75090802c85162daf687f41f182c11ecf75619d743acf7bb51"}
Sep 05 02:31:12.951993 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:12.951567    2566 scope.go:117] "RemoveContainer" containerID="0786ab948341ff1c5ddfbfe376b773716959ef1d894a3fb5de590ceccd580674"
Sep 05 02:31:12.959451 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:12.959435    2566 scope.go:117] "RemoveContainer" containerID="54fbf9ce2c8217226e5cb050f3cac1a453f288c8200720551d76e9d309fedc0a"
Sep 05 02:31:12.971223 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:12.971201    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"]
Sep 05 02:31:12.974436 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:12.974415    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a6d98-predictor-697b88f8b7-9wscs"]
Sep 05 02:31:13.047266 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:13.047243    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5653e59-452f-470e-b514-988ffc7c106c" path="/var/lib/kubelet/pods/f5653e59-452f-470e-b514-988ffc7c106c/volumes"
Sep 05 02:31:14.042814 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:14.042766    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 05 02:31:16.956025 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:16.955995    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"
Sep 05 02:31:16.956566 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:16.956531    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc" podUID="44f9ca6b-951b-4716-b11a-2c8d478c9a95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 05 02:31:17.201552 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:17.201513    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:24.043577 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:24.043539    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:31:26.957060 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:26.957015    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc" podUID="44f9ca6b-951b-4716-b11a-2c8d478c9a95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 05 02:31:36.957241 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:36.957199    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc" podUID="44f9ca6b-951b-4716-b11a-2c8d478c9a95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 05 02:31:46.956957 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:46.956917    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc" podUID="44f9ca6b-951b-4716-b11a-2c8d478c9a95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 05 02:31:48.470506 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:48.470465    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:48.602431 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:48.602397    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:56.957754 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:56.957722    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"
Sep 05 02:31:58.590563 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:58.590525    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:58.662862 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:58.662808    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"]
Sep 05 02:31:58.663141 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:58.663108    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="kserve-container" containerID="cri-o://630ea1d77c830483e8e54bcf1e51bcf760794a5af1bc7aca03f894ebee14bd25" gracePeriod=30
Sep 05 02:31:58.663288 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:58.663131    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="kube-rbac-proxy" containerID="cri-o://acacf73dff8ff16edd0d3ac2b63c029d707f7b8e2b75006a03cbc116b17a0bf4" gracePeriod=30
Sep 05 02:31:58.703748 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:58.703713    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:58.805265 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:58.805234    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:58.906078 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:58.906044    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:58.913084 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:58.913059    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"]
Sep 05 02:31:58.913394 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:58.913380    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5653e59-452f-470e-b514-988ffc7c106c" containerName="kube-rbac-proxy"
Sep 05 02:31:58.913439 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:58.913396    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5653e59-452f-470e-b514-988ffc7c106c" containerName="kube-rbac-proxy"
Sep 05 02:31:58.913439 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:58.913406    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5653e59-452f-470e-b514-988ffc7c106c" containerName="kserve-container"
Sep 05 02:31:58.913439 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:58.913411    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5653e59-452f-470e-b514-988ffc7c106c" containerName="kserve-container"
Sep 05 02:31:58.913527 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:58.913462    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5653e59-452f-470e-b514-988ffc7c106c" containerName="kube-rbac-proxy"
Sep 05 02:31:58.913527 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:58.913472    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5653e59-452f-470e-b514-988ffc7c106c" containerName="kserve-container"
Sep 05 02:31:58.916282 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:58.916268    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"
Sep 05 02:31:58.919069 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:58.919030    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9f825-predictor-serving-cert\""
Sep 05 02:31:58.919363 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:58.919341    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9f825-kube-rbac-proxy-sar-config\""
Sep 05 02:31:58.925597 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:58.925574    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"]
Sep 05 02:31:59.003801 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:59.003765    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:59.020845 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:59.020808    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/385b087b-b7c4-44fe-ba44-d7db73de0b18-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9f825-predictor-847f658579-n7z45\" (UID: \"385b087b-b7c4-44fe-ba44-d7db73de0b18\") " pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"
Sep 05 02:31:59.021010 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:59.020885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwszw\" (UniqueName: \"kubernetes.io/projected/385b087b-b7c4-44fe-ba44-d7db73de0b18-kube-api-access-qwszw\") pod \"error-404-isvc-9f825-predictor-847f658579-n7z45\" (UID: \"385b087b-b7c4-44fe-ba44-d7db73de0b18\") " pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"
Sep 05 02:31:59.021010 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:59.020977    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/385b087b-b7c4-44fe-ba44-d7db73de0b18-proxy-tls\") pod \"error-404-isvc-9f825-predictor-847f658579-n7z45\" (UID: \"385b087b-b7c4-44fe-ba44-d7db73de0b18\") " pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"
Sep 05 02:31:59.105066 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:59.105022    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:59.121627 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:59.121602    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/385b087b-b7c4-44fe-ba44-d7db73de0b18-proxy-tls\") pod \"error-404-isvc-9f825-predictor-847f658579-n7z45\" (UID: \"385b087b-b7c4-44fe-ba44-d7db73de0b18\") " pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"
Sep 05 02:31:59.121764 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:59.121640    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/385b087b-b7c4-44fe-ba44-d7db73de0b18-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9f825-predictor-847f658579-n7z45\" (UID: \"385b087b-b7c4-44fe-ba44-d7db73de0b18\") " pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"
Sep 05 02:31:59.121764 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:59.121675    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwszw\" (UniqueName: \"kubernetes.io/projected/385b087b-b7c4-44fe-ba44-d7db73de0b18-kube-api-access-qwszw\") pod \"error-404-isvc-9f825-predictor-847f658579-n7z45\" (UID: \"385b087b-b7c4-44fe-ba44-d7db73de0b18\") " pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"
Sep 05 02:31:59.122260 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:59.122240    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/385b087b-b7c4-44fe-ba44-d7db73de0b18-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9f825-predictor-847f658579-n7z45\" (UID: \"385b087b-b7c4-44fe-ba44-d7db73de0b18\") " pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"
Sep 05 02:31:59.124050 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:59.124028    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/385b087b-b7c4-44fe-ba44-d7db73de0b18-proxy-tls\") pod \"error-404-isvc-9f825-predictor-847f658579-n7z45\" (UID: \"385b087b-b7c4-44fe-ba44-d7db73de0b18\") " pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"
Sep 05 02:31:59.130612 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:59.130590    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwszw\" (UniqueName: \"kubernetes.io/projected/385b087b-b7c4-44fe-ba44-d7db73de0b18-kube-api-access-qwszw\") pod \"error-404-isvc-9f825-predictor-847f658579-n7z45\" (UID: \"385b087b-b7c4-44fe-ba44-d7db73de0b18\") " pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"
Sep 05 02:31:59.139516 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:59.139491    2566 generic.go:356] "Generic (PLEG): container finished" podID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerID="acacf73dff8ff16edd0d3ac2b63c029d707f7b8e2b75006a03cbc116b17a0bf4" exitCode=2
Sep 05 02:31:59.139616 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:59.139557    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" event={"ID":"fe25f418-9579-48f2-b9da-cd28e490c3b5","Type":"ContainerDied","Data":"acacf73dff8ff16edd0d3ac2b63c029d707f7b8e2b75006a03cbc116b17a0bf4"}
Sep 05 02:31:59.201993 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:59.201906    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:59.226958 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:59.226917    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"
Sep 05 02:31:59.497128 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:31:59.496924    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"]
Sep 05 02:31:59.499763 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:31:59.499730    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod385b087b_b7c4_44fe_ba44_d7db73de0b18.slice/crio-b24be38c60318eaeaafc2ea0fc4370c88a868b26c3e65f22e8fb52d936bbb7f7 WatchSource:0}: Error finding container b24be38c60318eaeaafc2ea0fc4370c88a868b26c3e65f22e8fb52d936bbb7f7: Status 404 returned error can't find the container with id b24be38c60318eaeaafc2ea0fc4370c88a868b26c3e65f22e8fb52d936bbb7f7
Sep 05 02:32:00.145717 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:00.145684    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45" event={"ID":"385b087b-b7c4-44fe-ba44-d7db73de0b18","Type":"ContainerStarted","Data":"dcccdc503492fe2e4c39682ad49715b84d9cca615640059ebb8cda2a6a61946b"}
Sep 05 02:32:00.146115 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:00.145738    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45" event={"ID":"385b087b-b7c4-44fe-ba44-d7db73de0b18","Type":"ContainerStarted","Data":"66e096973e9b5f21f140f9dbe18e588a5b2fb351069da36c45a42dc1fc1b1476"}
Sep 05 02:32:00.146115 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:00.145766    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45" event={"ID":"385b087b-b7c4-44fe-ba44-d7db73de0b18","Type":"ContainerStarted","Data":"b24be38c60318eaeaafc2ea0fc4370c88a868b26c3e65f22e8fb52d936bbb7f7"}
Sep 05 02:32:00.146115 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:00.145854    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"
Sep 05 02:32:00.163296 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:00.163252    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45" podStartSLOduration=2.163237285 podStartE2EDuration="2.163237285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:32:00.161793912 +0000 UTC m=+851.641188069" watchObservedRunningTime="2026-09-05 02:32:00.163237285 +0000 UTC m=+851.642631442"
Sep 05 02:32:00.700164 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:00.700122    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 05 02:32:01.149444 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:01.149413    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"
Sep 05 02:32:01.150738 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:01.150706    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45" podUID="385b087b-b7c4-44fe-ba44-d7db73de0b18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 05 02:32:02.155745 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:02.155705    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45" podUID="385b087b-b7c4-44fe-ba44-d7db73de0b18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 05 02:32:04.043463 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:04.043415    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 05 02:32:05.699493 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:05.699445    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 05 02:32:06.416132 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:06.416104    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:32:06.577984 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:06.577893    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe25f418-9579-48f2-b9da-cd28e490c3b5-proxy-tls\") pod \"fe25f418-9579-48f2-b9da-cd28e490c3b5\" (UID: \"fe25f418-9579-48f2-b9da-cd28e490c3b5\") "
Sep 05 02:32:06.577984 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:06.577984    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8zfz\" (UniqueName: \"kubernetes.io/projected/fe25f418-9579-48f2-b9da-cd28e490c3b5-kube-api-access-s8zfz\") pod \"fe25f418-9579-48f2-b9da-cd28e490c3b5\" (UID: \"fe25f418-9579-48f2-b9da-cd28e490c3b5\") "
Sep 05 02:32:06.578203 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:06.578012    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe25f418-9579-48f2-b9da-cd28e490c3b5-kserve-provision-location\") pod \"fe25f418-9579-48f2-b9da-cd28e490c3b5\" (UID: \"fe25f418-9579-48f2-b9da-cd28e490c3b5\") "
Sep 05 02:32:06.578203 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:06.578045    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe25f418-9579-48f2-b9da-cd28e490c3b5-kube-rbac-proxy-sar-config\") pod \"fe25f418-9579-48f2-b9da-cd28e490c3b5\" (UID: \"fe25f418-9579-48f2-b9da-cd28e490c3b5\") "
Sep 05 02:32:06.578203 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:06.578083    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe25f418-9579-48f2-b9da-cd28e490c3b5-cabundle-cert\") pod \"fe25f418-9579-48f2-b9da-cd28e490c3b5\" (UID: \"fe25f418-9579-48f2-b9da-cd28e490c3b5\") "
Sep 05 02:32:06.578364 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:06.578332    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe25f418-9579-48f2-b9da-cd28e490c3b5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fe25f418-9579-48f2-b9da-cd28e490c3b5" (UID: "fe25f418-9579-48f2-b9da-cd28e490c3b5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:32:06.578445 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:06.578423    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe25f418-9579-48f2-b9da-cd28e490c3b5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fe25f418-9579-48f2-b9da-cd28e490c3b5" (UID: "fe25f418-9579-48f2-b9da-cd28e490c3b5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:32:06.578504 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:06.578487    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe25f418-9579-48f2-b9da-cd28e490c3b5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fe25f418-9579-48f2-b9da-cd28e490c3b5" (UID: "fe25f418-9579-48f2-b9da-cd28e490c3b5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:32:06.580080 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:06.580056    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe25f418-9579-48f2-b9da-cd28e490c3b5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fe25f418-9579-48f2-b9da-cd28e490c3b5" (UID: "fe25f418-9579-48f2-b9da-cd28e490c3b5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:32:06.580189 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:06.580088    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe25f418-9579-48f2-b9da-cd28e490c3b5-kube-api-access-s8zfz" (OuterVolumeSpecName: "kube-api-access-s8zfz") pod "fe25f418-9579-48f2-b9da-cd28e490c3b5" (UID: "fe25f418-9579-48f2-b9da-cd28e490c3b5"). InnerVolumeSpecName "kube-api-access-s8zfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:32:06.678661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:06.678627    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8zfz\" (UniqueName: \"kubernetes.io/projected/fe25f418-9579-48f2-b9da-cd28e490c3b5-kube-api-access-s8zfz\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:32:06.678661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:06.678654    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe25f418-9579-48f2-b9da-cd28e490c3b5-kserve-provision-location\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:32:06.678661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:06.678663    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe25f418-9579-48f2-b9da-cd28e490c3b5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:32:06.678934 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:06.678672    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe25f418-9579-48f2-b9da-cd28e490c3b5-cabundle-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:32:06.678934 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:06.678682    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe25f418-9579-48f2-b9da-cd28e490c3b5-proxy-tls\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:32:07.160712 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:07.160683    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"
Sep 05 02:32:07.161261 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:07.161234    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45" podUID="385b087b-b7c4-44fe-ba44-d7db73de0b18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 05 02:32:07.176696 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:07.176664    2566 generic.go:356] "Generic (PLEG): container finished" podID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerID="630ea1d77c830483e8e54bcf1e51bcf760794a5af1bc7aca03f894ebee14bd25" exitCode=0
Sep 05 02:32:07.176866 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:07.176736    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"
Sep 05 02:32:07.176866 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:07.176746    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" event={"ID":"fe25f418-9579-48f2-b9da-cd28e490c3b5","Type":"ContainerDied","Data":"630ea1d77c830483e8e54bcf1e51bcf760794a5af1bc7aca03f894ebee14bd25"}
Sep 05 02:32:07.176866 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:07.176801    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj" event={"ID":"fe25f418-9579-48f2-b9da-cd28e490c3b5","Type":"ContainerDied","Data":"735aba264dff81b6ac191dd5c5f084185d698ba67cd1950aab6487ad79335988"}
Sep 05 02:32:07.176866 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:07.176853    2566 scope.go:117] "RemoveContainer" containerID="acacf73dff8ff16edd0d3ac2b63c029d707f7b8e2b75006a03cbc116b17a0bf4"
Sep 05 02:32:07.185399 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:07.185379    2566 scope.go:117] "RemoveContainer" containerID="630ea1d77c830483e8e54bcf1e51bcf760794a5af1bc7aca03f894ebee14bd25"
Sep 05 02:32:07.192662 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:07.192644    2566 scope.go:117] "RemoveContainer" containerID="a372c8aeef69be38962bdf52736b49fc04ef4f44529fd7407a4754ef6e81a2dc"
Sep 05 02:32:07.195123 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:07.195096    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"]
Sep 05 02:32:07.199624 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:07.199600    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-f5c56b76d-td6qj"]
Sep 05 02:32:07.200649 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:07.200628    2566 scope.go:117] "RemoveContainer" containerID="acacf73dff8ff16edd0d3ac2b63c029d707f7b8e2b75006a03cbc116b17a0bf4"
Sep 05 02:32:07.200954 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:32:07.200936    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acacf73dff8ff16edd0d3ac2b63c029d707f7b8e2b75006a03cbc116b17a0bf4\": container with ID starting with acacf73dff8ff16edd0d3ac2b63c029d707f7b8e2b75006a03cbc116b17a0bf4 not found: ID does not exist" containerID="acacf73dff8ff16edd0d3ac2b63c029d707f7b8e2b75006a03cbc116b17a0bf4"
Sep 05 02:32:07.201006 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:07.200962    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acacf73dff8ff16edd0d3ac2b63c029d707f7b8e2b75006a03cbc116b17a0bf4"} err="failed to get container status \"acacf73dff8ff16edd0d3ac2b63c029d707f7b8e2b75006a03cbc116b17a0bf4\": rpc error: code = NotFound desc = could not find container \"acacf73dff8ff16edd0d3ac2b63c029d707f7b8e2b75006a03cbc116b17a0bf4\": container with ID starting with acacf73dff8ff16edd0d3ac2b63c029d707f7b8e2b75006a03cbc116b17a0bf4 not found: ID does not exist"
Sep 05 02:32:07.201006 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:07.200985    2566 scope.go:117] "RemoveContainer" containerID="630ea1d77c830483e8e54bcf1e51bcf760794a5af1bc7aca03f894ebee14bd25"
Sep 05 02:32:07.201221 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:32:07.201203    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"630ea1d77c830483e8e54bcf1e51bcf760794a5af1bc7aca03f894ebee14bd25\": container with ID starting with 630ea1d77c830483e8e54bcf1e51bcf760794a5af1bc7aca03f894ebee14bd25 not found: ID does not exist" containerID="630ea1d77c830483e8e54bcf1e51bcf760794a5af1bc7aca03f894ebee14bd25"
Sep 05 02:32:07.201289 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:07.201232    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"630ea1d77c830483e8e54bcf1e51bcf760794a5af1bc7aca03f894ebee14bd25"} err="failed to get container status \"630ea1d77c830483e8e54bcf1e51bcf760794a5af1bc7aca03f894ebee14bd25\": rpc error: code = NotFound desc = could not find container \"630ea1d77c830483e8e54bcf1e51bcf760794a5af1bc7aca03f894ebee14bd25\": container with ID starting with 630ea1d77c830483e8e54bcf1e51bcf760794a5af1bc7aca03f894ebee14bd25 not found: ID does not exist"
Sep 05 02:32:07.201289 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:07.201256    2566 scope.go:117] "RemoveContainer" containerID="a372c8aeef69be38962bdf52736b49fc04ef4f44529fd7407a4754ef6e81a2dc"
Sep 05 02:32:07.201505 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:32:07.201489    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a372c8aeef69be38962bdf52736b49fc04ef4f44529fd7407a4754ef6e81a2dc\": container with ID starting with a372c8aeef69be38962bdf52736b49fc04ef4f44529fd7407a4754ef6e81a2dc not found: ID does not exist" containerID="a372c8aeef69be38962bdf52736b49fc04ef4f44529fd7407a4754ef6e81a2dc"
Sep 05 02:32:07.201543 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:07.201508    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a372c8aeef69be38962bdf52736b49fc04ef4f44529fd7407a4754ef6e81a2dc"} err="failed to get container status \"a372c8aeef69be38962bdf52736b49fc04ef4f44529fd7407a4754ef6e81a2dc\": rpc error: code = NotFound desc = could not find container \"a372c8aeef69be38962bdf52736b49fc04ef4f44529fd7407a4754ef6e81a2dc\": container with ID starting with a372c8aeef69be38962bdf52736b49fc04ef4f44529fd7407a4754ef6e81a2dc not found: ID does not exist"
Sep 05 02:32:08.934806 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:08.934772    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:32:09.048183 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:09.048152    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" path="/var/lib/kubelet/pods/fe25f418-9579-48f2-b9da-cd28e490c3b5/volumes"
Sep 05 02:32:09.158683 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:09.158654    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:32:17.162092 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:17.162049    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45" podUID="385b087b-b7c4-44fe-ba44-d7db73de0b18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 05 02:32:27.161803 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:27.161756    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45" podUID="385b087b-b7c4-44fe-ba44-d7db73de0b18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 05 02:32:37.161653 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:37.161611    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45" podUID="385b087b-b7c4-44fe-ba44-d7db73de0b18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 05 02:32:47.162710 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:47.162636    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"
Sep 05 02:32:58.775418 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:58.775383    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:32:59.173208 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:32:59.173175    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:38.635161 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:36:38.635115    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:37:51.192094 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:37:51.191990    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:37:51.255742 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:37:51.255700    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:37:58.977782 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:37:58.977743    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:37:59.053190 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:37:59.053160    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:05.886650 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:38:05.886618    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:05.988610 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:38:05.988566    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:39:05.761039 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:39:05.760991    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:40:23.600441 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.600352    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:40:23.715886 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.715851    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:40:23.798911 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.798880    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:40:23.807914 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.807882    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"]
Sep 05 02:40:23.808176 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.808142    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc" podUID="44f9ca6b-951b-4716-b11a-2c8d478c9a95" containerName="kserve-container" containerID="cri-o://12638a228971c5a8a28dfeedfac207790534dc095d0bf8a869667dd787910f09" gracePeriod=30
Sep 05 02:40:23.808515 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.808384    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc" podUID="44f9ca6b-951b-4716-b11a-2c8d478c9a95" containerName="kube-rbac-proxy" containerID="cri-o://bb3a0fd055c590cfc3e33f65eb8cf3db605dbd68d32882cf0b43b075a71b355b" gracePeriod=30
Sep 05 02:40:23.857610 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.857522    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"]
Sep 05 02:40:23.857918 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.857899    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="storage-initializer"
Sep 05 02:40:23.858012 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.857921    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="storage-initializer"
Sep 05 02:40:23.858012 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.857931    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="kserve-container"
Sep 05 02:40:23.858012 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.857939    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="kserve-container"
Sep 05 02:40:23.858012 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.857956    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="kube-rbac-proxy"
Sep 05 02:40:23.858012 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.857965    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="kube-rbac-proxy"
Sep 05 02:40:23.858253 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.858046    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="kserve-container"
Sep 05 02:40:23.858253 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.858058    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe25f418-9579-48f2-b9da-cd28e490c3b5" containerName="kube-rbac-proxy"
Sep 05 02:40:23.861464 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.861441    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"
Sep 05 02:40:23.863644 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.863622    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-12d88-kube-rbac-proxy-sar-config\""
Sep 05 02:40:23.863751 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.863672    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-12d88-predictor-serving-cert\""
Sep 05 02:40:23.871790 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.871768    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"]
Sep 05 02:40:23.872264 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.872244    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:40:23.890513 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.890475    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/175df11e-738d-491f-b1b7-5f66190b764a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-12d88-predictor-55c899946f-kjljc\" (UID: \"175df11e-738d-491f-b1b7-5f66190b764a\") " pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"
Sep 05 02:40:23.890643 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.890559    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx5c5\" (UniqueName: \"kubernetes.io/projected/175df11e-738d-491f-b1b7-5f66190b764a-kube-api-access-fx5c5\") pod \"error-404-isvc-12d88-predictor-55c899946f-kjljc\" (UID: \"175df11e-738d-491f-b1b7-5f66190b764a\") " pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"
Sep 05 02:40:23.890643 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.890604    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175df11e-738d-491f-b1b7-5f66190b764a-proxy-tls\") pod \"error-404-isvc-12d88-predictor-55c899946f-kjljc\" (UID: \"175df11e-738d-491f-b1b7-5f66190b764a\") " pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"
Sep 05 02:40:23.980043 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.979998    2566 generic.go:356] "Generic (PLEG): container finished" podID="44f9ca6b-951b-4716-b11a-2c8d478c9a95" containerID="bb3a0fd055c590cfc3e33f65eb8cf3db605dbd68d32882cf0b43b075a71b355b" exitCode=2
Sep 05 02:40:23.980195 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.980051    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc" event={"ID":"44f9ca6b-951b-4716-b11a-2c8d478c9a95","Type":"ContainerDied","Data":"bb3a0fd055c590cfc3e33f65eb8cf3db605dbd68d32882cf0b43b075a71b355b"}
Sep 05 02:40:23.991381 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.991345    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/175df11e-738d-491f-b1b7-5f66190b764a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-12d88-predictor-55c899946f-kjljc\" (UID: \"175df11e-738d-491f-b1b7-5f66190b764a\") " pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"
Sep 05 02:40:23.991561 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.991402    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx5c5\" (UniqueName: \"kubernetes.io/projected/175df11e-738d-491f-b1b7-5f66190b764a-kube-api-access-fx5c5\") pod \"error-404-isvc-12d88-predictor-55c899946f-kjljc\" (UID: \"175df11e-738d-491f-b1b7-5f66190b764a\") " pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"
Sep 05 02:40:23.991561 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.991441    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175df11e-738d-491f-b1b7-5f66190b764a-proxy-tls\") pod \"error-404-isvc-12d88-predictor-55c899946f-kjljc\" (UID: \"175df11e-738d-491f-b1b7-5f66190b764a\") " pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"
Sep 05 02:40:23.991674 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:40:23.991573    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-12d88-predictor-serving-cert: secret "error-404-isvc-12d88-predictor-serving-cert" not found
Sep 05 02:40:23.991674 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:40:23.991642    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/175df11e-738d-491f-b1b7-5f66190b764a-proxy-tls podName:175df11e-738d-491f-b1b7-5f66190b764a nodeName:}" failed. No retries permitted until 2026-09-05 02:40:24.491621204 +0000 UTC m=+1355.971015350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/175df11e-738d-491f-b1b7-5f66190b764a-proxy-tls") pod "error-404-isvc-12d88-predictor-55c899946f-kjljc" (UID: "175df11e-738d-491f-b1b7-5f66190b764a") : secret "error-404-isvc-12d88-predictor-serving-cert" not found
Sep 05 02:40:23.992054 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:23.992034    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/175df11e-738d-491f-b1b7-5f66190b764a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-12d88-predictor-55c899946f-kjljc\" (UID: \"175df11e-738d-491f-b1b7-5f66190b764a\") " pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"
Sep 05 02:40:24.003722 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:24.003692    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx5c5\" (UniqueName: \"kubernetes.io/projected/175df11e-738d-491f-b1b7-5f66190b764a-kube-api-access-fx5c5\") pod \"error-404-isvc-12d88-predictor-55c899946f-kjljc\" (UID: \"175df11e-738d-491f-b1b7-5f66190b764a\") " pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"
Sep 05 02:40:24.063029 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:24.062988    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:40:24.163156 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:24.163126    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:40:24.495909 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:24.495804    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175df11e-738d-491f-b1b7-5f66190b764a-proxy-tls\") pod \"error-404-isvc-12d88-predictor-55c899946f-kjljc\" (UID: \"175df11e-738d-491f-b1b7-5f66190b764a\") " pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"
Sep 05 02:40:24.498133 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:24.498111    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175df11e-738d-491f-b1b7-5f66190b764a-proxy-tls\") pod \"error-404-isvc-12d88-predictor-55c899946f-kjljc\" (UID: \"175df11e-738d-491f-b1b7-5f66190b764a\") " pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"
Sep 05 02:40:24.771757 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:24.771678    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"
Sep 05 02:40:25.041451 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:25.041385    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"]
Sep 05 02:40:25.043846 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:40:25.043805    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175df11e_738d_491f_b1b7_5f66190b764a.slice/crio-665c0484b5978e5b12fd4958281081140c2b455553ca7c33edcb7110386fb3f3 WatchSource:0}: Error finding container 665c0484b5978e5b12fd4958281081140c2b455553ca7c33edcb7110386fb3f3: Status 404 returned error can't find the container with id 665c0484b5978e5b12fd4958281081140c2b455553ca7c33edcb7110386fb3f3
Sep 05 02:40:25.992106 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:25.992063    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc" event={"ID":"175df11e-738d-491f-b1b7-5f66190b764a","Type":"ContainerStarted","Data":"b32011aa894b85aa1426dffc5ed4bb51ea71c017912ce9fbaa05ef393d45f044"}
Sep 05 02:40:25.992546 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:25.992158    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"
Sep 05 02:40:25.992546 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:25.992188    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc" event={"ID":"175df11e-738d-491f-b1b7-5f66190b764a","Type":"ContainerStarted","Data":"9042494398d1c6195152445cdab6c5f5cbc44029b190369253352c5f99d7e497"}
Sep 05 02:40:25.992546 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:25.992212    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc" event={"ID":"175df11e-738d-491f-b1b7-5f66190b764a","Type":"ContainerStarted","Data":"665c0484b5978e5b12fd4958281081140c2b455553ca7c33edcb7110386fb3f3"}
Sep 05 02:40:25.992546 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:25.992231    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"
Sep 05 02:40:25.993468 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:25.993444    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc" podUID="175df11e-738d-491f-b1b7-5f66190b764a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:40:26.012778 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:26.012716    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc" podStartSLOduration=3.012700121 podStartE2EDuration="3.012700121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:40:26.011202203 +0000 UTC m=+1357.490596365" watchObservedRunningTime="2026-09-05 02:40:26.012700121 +0000 UTC m=+1357.492094279"
Sep 05 02:40:26.947237 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:26.947190    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc" podUID="44f9ca6b-951b-4716-b11a-2c8d478c9a95" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 05 02:40:26.957417 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:26.957385    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc" podUID="44f9ca6b-951b-4716-b11a-2c8d478c9a95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 05 02:40:26.998404 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:26.998378    2566 generic.go:356] "Generic (PLEG): container finished" podID="44f9ca6b-951b-4716-b11a-2c8d478c9a95" containerID="12638a228971c5a8a28dfeedfac207790534dc095d0bf8a869667dd787910f09" exitCode=0
Sep 05 02:40:26.998745 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:26.998445    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc" event={"ID":"44f9ca6b-951b-4716-b11a-2c8d478c9a95","Type":"ContainerDied","Data":"12638a228971c5a8a28dfeedfac207790534dc095d0bf8a869667dd787910f09"}
Sep 05 02:40:26.998745 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:26.998713    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc" podUID="175df11e-738d-491f-b1b7-5f66190b764a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:40:27.051137 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:27.051107    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"
Sep 05 02:40:27.115398 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:27.115367    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44f9ca6b-951b-4716-b11a-2c8d478c9a95-kube-rbac-proxy-sar-config\") pod \"44f9ca6b-951b-4716-b11a-2c8d478c9a95\" (UID: \"44f9ca6b-951b-4716-b11a-2c8d478c9a95\") "
Sep 05 02:40:27.115398 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:27.115403    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqmpp\" (UniqueName: \"kubernetes.io/projected/44f9ca6b-951b-4716-b11a-2c8d478c9a95-kube-api-access-rqmpp\") pod \"44f9ca6b-951b-4716-b11a-2c8d478c9a95\" (UID: \"44f9ca6b-951b-4716-b11a-2c8d478c9a95\") "
Sep 05 02:40:27.115649 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:27.115427    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44f9ca6b-951b-4716-b11a-2c8d478c9a95-proxy-tls\") pod \"44f9ca6b-951b-4716-b11a-2c8d478c9a95\" (UID: \"44f9ca6b-951b-4716-b11a-2c8d478c9a95\") "
Sep 05 02:40:27.116253 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:27.115778    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f9ca6b-951b-4716-b11a-2c8d478c9a95-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "44f9ca6b-951b-4716-b11a-2c8d478c9a95" (UID: "44f9ca6b-951b-4716-b11a-2c8d478c9a95"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:40:27.117642 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:27.117622    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f9ca6b-951b-4716-b11a-2c8d478c9a95-kube-api-access-rqmpp" (OuterVolumeSpecName: "kube-api-access-rqmpp") pod "44f9ca6b-951b-4716-b11a-2c8d478c9a95" (UID: "44f9ca6b-951b-4716-b11a-2c8d478c9a95"). InnerVolumeSpecName "kube-api-access-rqmpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:40:27.117716 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:27.117650    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44f9ca6b-951b-4716-b11a-2c8d478c9a95-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "44f9ca6b-951b-4716-b11a-2c8d478c9a95" (UID: "44f9ca6b-951b-4716-b11a-2c8d478c9a95"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:40:27.217174 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:27.217100    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44f9ca6b-951b-4716-b11a-2c8d478c9a95-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:40:27.217174 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:27.217123    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqmpp\" (UniqueName: \"kubernetes.io/projected/44f9ca6b-951b-4716-b11a-2c8d478c9a95-kube-api-access-rqmpp\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:40:27.217174 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:27.217132    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44f9ca6b-951b-4716-b11a-2c8d478c9a95-proxy-tls\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:40:28.004008 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:28.003965    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc" event={"ID":"44f9ca6b-951b-4716-b11a-2c8d478c9a95","Type":"ContainerDied","Data":"4e42e9a68bbc4471bbecd0a6457450ec41b54d898b8621c057466ee61d8e9476"}
Sep 05 02:40:28.004008 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:28.004003    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"
Sep 05 02:40:28.004438 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:28.004053    2566 scope.go:117] "RemoveContainer" containerID="bb3a0fd055c590cfc3e33f65eb8cf3db605dbd68d32882cf0b43b075a71b355b"
Sep 05 02:40:28.013401 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:28.013379    2566 scope.go:117] "RemoveContainer" containerID="12638a228971c5a8a28dfeedfac207790534dc095d0bf8a869667dd787910f09"
Sep 05 02:40:28.025250 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:28.025223    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"]
Sep 05 02:40:28.028786 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:28.028767    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3f635-predictor-bd67df8df-mxddc"]
Sep 05 02:40:29.047184 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:29.047151    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44f9ca6b-951b-4716-b11a-2c8d478c9a95" path="/var/lib/kubelet/pods/44f9ca6b-951b-4716-b11a-2c8d478c9a95/volumes"
Sep 05 02:40:32.008617 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:32.008585    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"
Sep 05 02:40:32.009182 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:32.009149    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc" podUID="175df11e-738d-491f-b1b7-5f66190b764a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:40:42.009570 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:42.009518    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc" podUID="175df11e-738d-491f-b1b7-5f66190b764a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:40:52.009541 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:40:52.009499    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc" podUID="175df11e-738d-491f-b1b7-5f66190b764a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:41:02.009142 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:02.009091    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc" podUID="175df11e-738d-491f-b1b7-5f66190b764a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 05 02:41:12.009988 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:12.009949    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"
Sep 05 02:41:13.499623 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.499586    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:13.599509 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.599467    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:13.604662 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.604630    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"]
Sep 05 02:41:13.604936 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.604911    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45" podUID="385b087b-b7c4-44fe-ba44-d7db73de0b18" containerName="kserve-container" containerID="cri-o://66e096973e9b5f21f140f9dbe18e588a5b2fb351069da36c45a42dc1fc1b1476" gracePeriod=30
Sep 05 02:41:13.605050 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.604934    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45" podUID="385b087b-b7c4-44fe-ba44-d7db73de0b18" containerName="kube-rbac-proxy" containerID="cri-o://dcccdc503492fe2e4c39682ad49715b84d9cca615640059ebb8cda2a6a61946b" gracePeriod=30
Sep 05 02:41:13.684683 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.684653    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:13.762331 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.762235    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:13.790815 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.790785    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"]
Sep 05 02:41:13.791091 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.791079    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44f9ca6b-951b-4716-b11a-2c8d478c9a95" containerName="kserve-container"
Sep 05 02:41:13.791137 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.791093    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="44f9ca6b-951b-4716-b11a-2c8d478c9a95" containerName="kserve-container"
Sep 05 02:41:13.791137 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.791104    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44f9ca6b-951b-4716-b11a-2c8d478c9a95" containerName="kube-rbac-proxy"
Sep 05 02:41:13.791137 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.791109    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="44f9ca6b-951b-4716-b11a-2c8d478c9a95" containerName="kube-rbac-proxy"
Sep 05 02:41:13.791226 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.791168    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="44f9ca6b-951b-4716-b11a-2c8d478c9a95" containerName="kserve-container"
Sep 05 02:41:13.791226 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.791177    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="44f9ca6b-951b-4716-b11a-2c8d478c9a95" containerName="kube-rbac-proxy"
Sep 05 02:41:13.794128 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.794111    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"
Sep 05 02:41:13.798504 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.798483    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-80c39-predictor-serving-cert\""
Sep 05 02:41:13.798661 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.798585    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-80c39-kube-rbac-proxy-sar-config\""
Sep 05 02:41:13.807946 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.807922    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"]
Sep 05 02:41:13.858713 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.858672    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:13.966974 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.966941    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:13.973947 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.973920    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb9f34f3-00f7-483a-87e1-7c8021003116-proxy-tls\") pod \"error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j\" (UID: \"cb9f34f3-00f7-483a-87e1-7c8021003116\") " pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"
Sep 05 02:41:13.974068 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.973965    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk4wf\" (UniqueName: \"kubernetes.io/projected/cb9f34f3-00f7-483a-87e1-7c8021003116-kube-api-access-tk4wf\") pod \"error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j\" (UID: \"cb9f34f3-00f7-483a-87e1-7c8021003116\") " pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"
Sep 05 02:41:13.974115 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:13.974065    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb9f34f3-00f7-483a-87e1-7c8021003116-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j\" (UID: \"cb9f34f3-00f7-483a-87e1-7c8021003116\") " pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"
Sep 05 02:41:14.075510 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:14.075425    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb9f34f3-00f7-483a-87e1-7c8021003116-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j\" (UID: \"cb9f34f3-00f7-483a-87e1-7c8021003116\") " pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"
Sep 05 02:41:14.075510 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:14.075474    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb9f34f3-00f7-483a-87e1-7c8021003116-proxy-tls\") pod \"error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j\" (UID: \"cb9f34f3-00f7-483a-87e1-7c8021003116\") " pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"
Sep 05 02:41:14.075699 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:14.075514    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk4wf\" (UniqueName: \"kubernetes.io/projected/cb9f34f3-00f7-483a-87e1-7c8021003116-kube-api-access-tk4wf\") pod \"error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j\" (UID: \"cb9f34f3-00f7-483a-87e1-7c8021003116\") " pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"
Sep 05 02:41:14.076196 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:14.076170    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb9f34f3-00f7-483a-87e1-7c8021003116-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j\" (UID: \"cb9f34f3-00f7-483a-87e1-7c8021003116\") " pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"
Sep 05 02:41:14.078004 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:14.077977    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb9f34f3-00f7-483a-87e1-7c8021003116-proxy-tls\") pod \"error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j\" (UID: \"cb9f34f3-00f7-483a-87e1-7c8021003116\") " pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"
Sep 05 02:41:14.083473 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:14.083453    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk4wf\" (UniqueName: \"kubernetes.io/projected/cb9f34f3-00f7-483a-87e1-7c8021003116-kube-api-access-tk4wf\") pod \"error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j\" (UID: \"cb9f34f3-00f7-483a-87e1-7c8021003116\") " pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"
Sep 05 02:41:14.104306 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:14.104279    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"
Sep 05 02:41:14.175190 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:14.175156    2566 generic.go:356] "Generic (PLEG): container finished" podID="385b087b-b7c4-44fe-ba44-d7db73de0b18" containerID="dcccdc503492fe2e4c39682ad49715b84d9cca615640059ebb8cda2a6a61946b" exitCode=2
Sep 05 02:41:14.175361 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:14.175237    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45" event={"ID":"385b087b-b7c4-44fe-ba44-d7db73de0b18","Type":"ContainerDied","Data":"dcccdc503492fe2e4c39682ad49715b84d9cca615640059ebb8cda2a6a61946b"}
Sep 05 02:41:14.376747 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:14.376714    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"]
Sep 05 02:41:14.379529 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:41:14.379500    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9f34f3_00f7_483a_87e1_7c8021003116.slice/crio-80c2811c4718c98cb1bc4a6efbd296100e2b73bc6460b73be37d009040cac15f WatchSource:0}: Error finding container 80c2811c4718c98cb1bc4a6efbd296100e2b73bc6460b73be37d009040cac15f: Status 404 returned error can't find the container with id 80c2811c4718c98cb1bc4a6efbd296100e2b73bc6460b73be37d009040cac15f
Sep 05 02:41:15.181098 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:15.181061    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j" event={"ID":"cb9f34f3-00f7-483a-87e1-7c8021003116","Type":"ContainerStarted","Data":"c90c43ec0f3c554f1a0766ea7e2b546a76e6dec81abbc80381b06410b7432498"}
Sep 05 02:41:15.181478 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:15.181119    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j" event={"ID":"cb9f34f3-00f7-483a-87e1-7c8021003116","Type":"ContainerStarted","Data":"1acfde1dfca685c284bea2b05c157dd35c0d291b1b2af49b205699b55fdacd81"}
Sep 05 02:41:15.181478 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:15.181142    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j" event={"ID":"cb9f34f3-00f7-483a-87e1-7c8021003116","Type":"ContainerStarted","Data":"80c2811c4718c98cb1bc4a6efbd296100e2b73bc6460b73be37d009040cac15f"}
Sep 05 02:41:15.181478 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:15.181165    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"
Sep 05 02:41:15.199864 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:15.199800    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j" podStartSLOduration=2.199785011 podStartE2EDuration="2.199785011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:41:15.198262794 +0000 UTC m=+1406.677656948" watchObservedRunningTime="2026-09-05 02:41:15.199785011 +0000 UTC m=+1406.679179166"
Sep 05 02:41:15.454804 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:15.454714    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:15.514656 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:15.514628    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:16.185312 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:16.185283    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"
Sep 05 02:41:16.186357 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:16.186331    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j" podUID="cb9f34f3-00f7-483a-87e1-7c8021003116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 05 02:41:16.941857 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:16.941819    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"
Sep 05 02:41:16.995758 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:16.995677    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwszw\" (UniqueName: \"kubernetes.io/projected/385b087b-b7c4-44fe-ba44-d7db73de0b18-kube-api-access-qwszw\") pod \"385b087b-b7c4-44fe-ba44-d7db73de0b18\" (UID: \"385b087b-b7c4-44fe-ba44-d7db73de0b18\") "
Sep 05 02:41:16.995758 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:16.995710    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/385b087b-b7c4-44fe-ba44-d7db73de0b18-proxy-tls\") pod \"385b087b-b7c4-44fe-ba44-d7db73de0b18\" (UID: \"385b087b-b7c4-44fe-ba44-d7db73de0b18\") "
Sep 05 02:41:16.995758 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:16.995731    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/385b087b-b7c4-44fe-ba44-d7db73de0b18-kube-rbac-proxy-sar-config\") pod \"385b087b-b7c4-44fe-ba44-d7db73de0b18\" (UID: \"385b087b-b7c4-44fe-ba44-d7db73de0b18\") "
Sep 05 02:41:16.996134 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:16.996108    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/385b087b-b7c4-44fe-ba44-d7db73de0b18-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "385b087b-b7c4-44fe-ba44-d7db73de0b18" (UID: "385b087b-b7c4-44fe-ba44-d7db73de0b18"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:41:16.997917 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:16.997886    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385b087b-b7c4-44fe-ba44-d7db73de0b18-kube-api-access-qwszw" (OuterVolumeSpecName: "kube-api-access-qwszw") pod "385b087b-b7c4-44fe-ba44-d7db73de0b18" (UID: "385b087b-b7c4-44fe-ba44-d7db73de0b18"). InnerVolumeSpecName "kube-api-access-qwszw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:41:16.997917 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:16.997896    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/385b087b-b7c4-44fe-ba44-d7db73de0b18-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "385b087b-b7c4-44fe-ba44-d7db73de0b18" (UID: "385b087b-b7c4-44fe-ba44-d7db73de0b18"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:41:17.097083 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:17.097042    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwszw\" (UniqueName: \"kubernetes.io/projected/385b087b-b7c4-44fe-ba44-d7db73de0b18-kube-api-access-qwszw\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:41:17.097083 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:17.097075    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/385b087b-b7c4-44fe-ba44-d7db73de0b18-proxy-tls\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:41:17.097083 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:17.097088    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/385b087b-b7c4-44fe-ba44-d7db73de0b18-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:41:17.190584 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:17.190550    2566 generic.go:356] "Generic (PLEG): container finished" podID="385b087b-b7c4-44fe-ba44-d7db73de0b18" containerID="66e096973e9b5f21f140f9dbe18e588a5b2fb351069da36c45a42dc1fc1b1476" exitCode=0
Sep 05 02:41:17.191035 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:17.190626    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45" event={"ID":"385b087b-b7c4-44fe-ba44-d7db73de0b18","Type":"ContainerDied","Data":"66e096973e9b5f21f140f9dbe18e588a5b2fb351069da36c45a42dc1fc1b1476"}
Sep 05 02:41:17.191035 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:17.190680    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45" event={"ID":"385b087b-b7c4-44fe-ba44-d7db73de0b18","Type":"ContainerDied","Data":"b24be38c60318eaeaafc2ea0fc4370c88a868b26c3e65f22e8fb52d936bbb7f7"}
Sep 05 02:41:17.191035 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:17.190707    2566 scope.go:117] "RemoveContainer" containerID="dcccdc503492fe2e4c39682ad49715b84d9cca615640059ebb8cda2a6a61946b"
Sep 05 02:41:17.191035 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:17.190630    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"
Sep 05 02:41:17.191252 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:17.191032    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j" podUID="cb9f34f3-00f7-483a-87e1-7c8021003116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 05 02:41:17.199095 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:17.199080    2566 scope.go:117] "RemoveContainer" containerID="66e096973e9b5f21f140f9dbe18e588a5b2fb351069da36c45a42dc1fc1b1476"
Sep 05 02:41:17.207538 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:17.207520    2566 scope.go:117] "RemoveContainer" containerID="dcccdc503492fe2e4c39682ad49715b84d9cca615640059ebb8cda2a6a61946b"
Sep 05 02:41:17.207854 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:41:17.207811    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcccdc503492fe2e4c39682ad49715b84d9cca615640059ebb8cda2a6a61946b\": container with ID starting with dcccdc503492fe2e4c39682ad49715b84d9cca615640059ebb8cda2a6a61946b not found: ID does not exist" containerID="dcccdc503492fe2e4c39682ad49715b84d9cca615640059ebb8cda2a6a61946b"
Sep 05 02:41:17.207971 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:17.207865    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcccdc503492fe2e4c39682ad49715b84d9cca615640059ebb8cda2a6a61946b"} err="failed to get container status \"dcccdc503492fe2e4c39682ad49715b84d9cca615640059ebb8cda2a6a61946b\": rpc error: code = NotFound desc = could not find container \"dcccdc503492fe2e4c39682ad49715b84d9cca615640059ebb8cda2a6a61946b\": container with ID starting with dcccdc503492fe2e4c39682ad49715b84d9cca615640059ebb8cda2a6a61946b not found: ID does not exist"
Sep 05 02:41:17.207971 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:17.207887    2566 scope.go:117] "RemoveContainer" containerID="66e096973e9b5f21f140f9dbe18e588a5b2fb351069da36c45a42dc1fc1b1476"
Sep 05 02:41:17.208286 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:41:17.208267    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66e096973e9b5f21f140f9dbe18e588a5b2fb351069da36c45a42dc1fc1b1476\": container with ID starting with 66e096973e9b5f21f140f9dbe18e588a5b2fb351069da36c45a42dc1fc1b1476 not found: ID does not exist" containerID="66e096973e9b5f21f140f9dbe18e588a5b2fb351069da36c45a42dc1fc1b1476"
Sep 05 02:41:17.208363 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:17.208290    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66e096973e9b5f21f140f9dbe18e588a5b2fb351069da36c45a42dc1fc1b1476"} err="failed to get container status \"66e096973e9b5f21f140f9dbe18e588a5b2fb351069da36c45a42dc1fc1b1476\": rpc error: code = NotFound desc = could not find container \"66e096973e9b5f21f140f9dbe18e588a5b2fb351069da36c45a42dc1fc1b1476\": container with ID starting with 66e096973e9b5f21f140f9dbe18e588a5b2fb351069da36c45a42dc1fc1b1476 not found: ID does not exist"
Sep 05 02:41:17.210365 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:17.210343    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"]
Sep 05 02:41:17.216450 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:17.216430    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9f825-predictor-847f658579-n7z45"]
Sep 05 02:41:19.047287 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:19.047257    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="385b087b-b7c4-44fe-ba44-d7db73de0b18" path="/var/lib/kubelet/pods/385b087b-b7c4-44fe-ba44-d7db73de0b18/volumes"
Sep 05 02:41:22.196197 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:22.196165    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"
Sep 05 02:41:22.196669 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:22.196639    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j" podUID="cb9f34f3-00f7-483a-87e1-7c8021003116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 05 02:41:23.808236 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:23.808198    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:23.902330 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:23.902293    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:32.197293 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:32.197257    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j" podUID="cb9f34f3-00f7-483a-87e1-7c8021003116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 05 02:41:33.886087 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:33.886055    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:33.978332 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:33.978297    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"]
Sep 05 02:41:33.978637 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:33.978605    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc" podUID="175df11e-738d-491f-b1b7-5f66190b764a" containerName="kserve-container" containerID="cri-o://9042494398d1c6195152445cdab6c5f5cbc44029b190369253352c5f99d7e497" gracePeriod=30
Sep 05 02:41:33.978749 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:33.978609    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc" podUID="175df11e-738d-491f-b1b7-5f66190b764a" containerName="kube-rbac-proxy" containerID="cri-o://b32011aa894b85aa1426dffc5ed4bb51ea71c017912ce9fbaa05ef393d45f044" gracePeriod=30
Sep 05 02:41:33.982624 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:33.982600    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:34.055644 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.055585    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:34.160386 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.160308    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:34.169607 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.169577    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"]
Sep 05 02:41:34.170015 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.169997    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="385b087b-b7c4-44fe-ba44-d7db73de0b18" containerName="kserve-container"
Sep 05 02:41:34.170015 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.170017    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="385b087b-b7c4-44fe-ba44-d7db73de0b18" containerName="kserve-container"
Sep 05 02:41:34.170177 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.170071    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="385b087b-b7c4-44fe-ba44-d7db73de0b18" containerName="kube-rbac-proxy"
Sep 05 02:41:34.170177 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.170083    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="385b087b-b7c4-44fe-ba44-d7db73de0b18" containerName="kube-rbac-proxy"
Sep 05 02:41:34.170271 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.170180    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="385b087b-b7c4-44fe-ba44-d7db73de0b18" containerName="kube-rbac-proxy"
Sep 05 02:41:34.170271 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.170191    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="385b087b-b7c4-44fe-ba44-d7db73de0b18" containerName="kserve-container"
Sep 05 02:41:34.173475 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.173450    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"
Sep 05 02:41:34.176056 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.176018    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9755b-predictor-serving-cert\""
Sep 05 02:41:34.176056 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.176034    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9755b-kube-rbac-proxy-sar-config\""
Sep 05 02:41:34.181544 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.181438    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"]
Sep 05 02:41:34.202766 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.202739    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06f8f0a4-d5d8-47ba-8716-67fe9f7d5597-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9755b-predictor-58df99b7-jlvhd\" (UID: \"06f8f0a4-d5d8-47ba-8716-67fe9f7d5597\") " pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"
Sep 05 02:41:34.202909 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.202780    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06f8f0a4-d5d8-47ba-8716-67fe9f7d5597-proxy-tls\") pod \"error-404-isvc-9755b-predictor-58df99b7-jlvhd\" (UID: \"06f8f0a4-d5d8-47ba-8716-67fe9f7d5597\") " pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"
Sep 05 02:41:34.202909 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.202799    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsdsc\" (UniqueName: \"kubernetes.io/projected/06f8f0a4-d5d8-47ba-8716-67fe9f7d5597-kube-api-access-xsdsc\") pod \"error-404-isvc-9755b-predictor-58df99b7-jlvhd\" (UID: \"06f8f0a4-d5d8-47ba-8716-67fe9f7d5597\") " pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"
Sep 05 02:41:34.255420 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.255388    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:34.256549 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.256522    2566 generic.go:356] "Generic (PLEG): container finished" podID="175df11e-738d-491f-b1b7-5f66190b764a" containerID="b32011aa894b85aa1426dffc5ed4bb51ea71c017912ce9fbaa05ef393d45f044" exitCode=2
Sep 05 02:41:34.256652 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.256559    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc" event={"ID":"175df11e-738d-491f-b1b7-5f66190b764a","Type":"ContainerDied","Data":"b32011aa894b85aa1426dffc5ed4bb51ea71c017912ce9fbaa05ef393d45f044"}
Sep 05 02:41:34.303361 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.303331    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06f8f0a4-d5d8-47ba-8716-67fe9f7d5597-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9755b-predictor-58df99b7-jlvhd\" (UID: \"06f8f0a4-d5d8-47ba-8716-67fe9f7d5597\") " pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"
Sep 05 02:41:34.303524 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.303374    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06f8f0a4-d5d8-47ba-8716-67fe9f7d5597-proxy-tls\") pod \"error-404-isvc-9755b-predictor-58df99b7-jlvhd\" (UID: \"06f8f0a4-d5d8-47ba-8716-67fe9f7d5597\") " pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"
Sep 05 02:41:34.303524 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.303391    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsdsc\" (UniqueName: \"kubernetes.io/projected/06f8f0a4-d5d8-47ba-8716-67fe9f7d5597-kube-api-access-xsdsc\") pod \"error-404-isvc-9755b-predictor-58df99b7-jlvhd\" (UID: \"06f8f0a4-d5d8-47ba-8716-67fe9f7d5597\") " pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"
Sep 05 02:41:34.304050 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.304025    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06f8f0a4-d5d8-47ba-8716-67fe9f7d5597-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9755b-predictor-58df99b7-jlvhd\" (UID: \"06f8f0a4-d5d8-47ba-8716-67fe9f7d5597\") " pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"
Sep 05 02:41:34.306010 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.305988    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06f8f0a4-d5d8-47ba-8716-67fe9f7d5597-proxy-tls\") pod \"error-404-isvc-9755b-predictor-58df99b7-jlvhd\" (UID: \"06f8f0a4-d5d8-47ba-8716-67fe9f7d5597\") " pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"
Sep 05 02:41:34.312846 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.312808    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsdsc\" (UniqueName: \"kubernetes.io/projected/06f8f0a4-d5d8-47ba-8716-67fe9f7d5597-kube-api-access-xsdsc\") pod \"error-404-isvc-9755b-predictor-58df99b7-jlvhd\" (UID: \"06f8f0a4-d5d8-47ba-8716-67fe9f7d5597\") " pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"
Sep 05 02:41:34.358226 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.358199    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:34.456675 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.456588    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:34.485630 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.485590    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"
Sep 05 02:41:34.757222 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:34.757145    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"]
Sep 05 02:41:34.759942 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:41:34.759912    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f8f0a4_d5d8_47ba_8716_67fe9f7d5597.slice/crio-ffdb61fce00ca1d925197c189617b310160fcec4e7f28df8f1b2183dd05aff96 WatchSource:0}: Error finding container ffdb61fce00ca1d925197c189617b310160fcec4e7f28df8f1b2183dd05aff96: Status 404 returned error can't find the container with id ffdb61fce00ca1d925197c189617b310160fcec4e7f28df8f1b2183dd05aff96
Sep 05 02:41:35.262022 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:35.261987    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd" event={"ID":"06f8f0a4-d5d8-47ba-8716-67fe9f7d5597","Type":"ContainerStarted","Data":"6ffaf25c8a20dab52293bf59f8e6d6d8a4f90bda52bfe02419cfc00a32cb5272"}
Sep 05 02:41:35.262428 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:35.262053    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd" event={"ID":"06f8f0a4-d5d8-47ba-8716-67fe9f7d5597","Type":"ContainerStarted","Data":"00143145a00e20a6627de96712c5dcf57dae0b2e769e2874576a504effed70d9"}
Sep 05 02:41:35.262428 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:35.262084    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd" event={"ID":"06f8f0a4-d5d8-47ba-8716-67fe9f7d5597","Type":"ContainerStarted","Data":"ffdb61fce00ca1d925197c189617b310160fcec4e7f28df8f1b2183dd05aff96"}
Sep 05 02:41:35.262428 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:35.262115    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"
Sep 05 02:41:35.282028 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:35.281978    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd" podStartSLOduration=1.281962212 podStartE2EDuration="1.281962212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:41:35.281034216 +0000 UTC m=+1426.760428372" watchObservedRunningTime="2026-09-05 02:41:35.281962212 +0000 UTC m=+1426.761356366"
Sep 05 02:41:36.265720 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:36.265688    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"
Sep 05 02:41:36.266964 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:36.266934    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd" podUID="06f8f0a4-d5d8-47ba-8716-67fe9f7d5597" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 05 02:41:36.999089 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:36.999048    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc" podUID="175df11e-738d-491f-b1b7-5f66190b764a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 05 02:41:37.224505 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.224478    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"
Sep 05 02:41:37.275755 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.275665    2566 generic.go:356] "Generic (PLEG): container finished" podID="175df11e-738d-491f-b1b7-5f66190b764a" containerID="9042494398d1c6195152445cdab6c5f5cbc44029b190369253352c5f99d7e497" exitCode=0
Sep 05 02:41:37.275755 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.275731    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"
Sep 05 02:41:37.276248 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.275751    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc" event={"ID":"175df11e-738d-491f-b1b7-5f66190b764a","Type":"ContainerDied","Data":"9042494398d1c6195152445cdab6c5f5cbc44029b190369253352c5f99d7e497"}
Sep 05 02:41:37.276248 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.275861    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc" event={"ID":"175df11e-738d-491f-b1b7-5f66190b764a","Type":"ContainerDied","Data":"665c0484b5978e5b12fd4958281081140c2b455553ca7c33edcb7110386fb3f3"}
Sep 05 02:41:37.276248 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.275900    2566 scope.go:117] "RemoveContainer" containerID="b32011aa894b85aa1426dffc5ed4bb51ea71c017912ce9fbaa05ef393d45f044"
Sep 05 02:41:37.276248 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.276156    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd" podUID="06f8f0a4-d5d8-47ba-8716-67fe9f7d5597" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 05 02:41:37.283723 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.283708    2566 scope.go:117] "RemoveContainer" containerID="9042494398d1c6195152445cdab6c5f5cbc44029b190369253352c5f99d7e497"
Sep 05 02:41:37.290629 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.290608    2566 scope.go:117] "RemoveContainer" containerID="b32011aa894b85aa1426dffc5ed4bb51ea71c017912ce9fbaa05ef393d45f044"
Sep 05 02:41:37.290933 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:41:37.290908    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b32011aa894b85aa1426dffc5ed4bb51ea71c017912ce9fbaa05ef393d45f044\": container with ID starting with b32011aa894b85aa1426dffc5ed4bb51ea71c017912ce9fbaa05ef393d45f044 not found: ID does not exist" containerID="b32011aa894b85aa1426dffc5ed4bb51ea71c017912ce9fbaa05ef393d45f044"
Sep 05 02:41:37.291038 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.290937    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b32011aa894b85aa1426dffc5ed4bb51ea71c017912ce9fbaa05ef393d45f044"} err="failed to get container status \"b32011aa894b85aa1426dffc5ed4bb51ea71c017912ce9fbaa05ef393d45f044\": rpc error: code = NotFound desc = could not find container \"b32011aa894b85aa1426dffc5ed4bb51ea71c017912ce9fbaa05ef393d45f044\": container with ID starting with b32011aa894b85aa1426dffc5ed4bb51ea71c017912ce9fbaa05ef393d45f044 not found: ID does not exist"
Sep 05 02:41:37.291038 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.290953    2566 scope.go:117] "RemoveContainer" containerID="9042494398d1c6195152445cdab6c5f5cbc44029b190369253352c5f99d7e497"
Sep 05 02:41:37.291210 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:41:37.291191    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9042494398d1c6195152445cdab6c5f5cbc44029b190369253352c5f99d7e497\": container with ID starting with 9042494398d1c6195152445cdab6c5f5cbc44029b190369253352c5f99d7e497 not found: ID does not exist" containerID="9042494398d1c6195152445cdab6c5f5cbc44029b190369253352c5f99d7e497"
Sep 05 02:41:37.291265 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.291219    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9042494398d1c6195152445cdab6c5f5cbc44029b190369253352c5f99d7e497"} err="failed to get container status \"9042494398d1c6195152445cdab6c5f5cbc44029b190369253352c5f99d7e497\": rpc error: code = NotFound desc = could not find container \"9042494398d1c6195152445cdab6c5f5cbc44029b190369253352c5f99d7e497\": container with ID starting with 9042494398d1c6195152445cdab6c5f5cbc44029b190369253352c5f99d7e497 not found: ID does not exist"
Sep 05 02:41:37.426177 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.426144    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fx5c5\" (UniqueName: \"kubernetes.io/projected/175df11e-738d-491f-b1b7-5f66190b764a-kube-api-access-fx5c5\") pod \"175df11e-738d-491f-b1b7-5f66190b764a\" (UID: \"175df11e-738d-491f-b1b7-5f66190b764a\") "
Sep 05 02:41:37.426177 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.426177    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175df11e-738d-491f-b1b7-5f66190b764a-proxy-tls\") pod \"175df11e-738d-491f-b1b7-5f66190b764a\" (UID: \"175df11e-738d-491f-b1b7-5f66190b764a\") "
Sep 05 02:41:37.426419 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.426201    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/175df11e-738d-491f-b1b7-5f66190b764a-kube-rbac-proxy-sar-config\") pod \"175df11e-738d-491f-b1b7-5f66190b764a\" (UID: \"175df11e-738d-491f-b1b7-5f66190b764a\") "
Sep 05 02:41:37.426552 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.426531    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175df11e-738d-491f-b1b7-5f66190b764a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "175df11e-738d-491f-b1b7-5f66190b764a" (UID: "175df11e-738d-491f-b1b7-5f66190b764a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:41:37.428333 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.428312    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/175df11e-738d-491f-b1b7-5f66190b764a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "175df11e-738d-491f-b1b7-5f66190b764a" (UID: "175df11e-738d-491f-b1b7-5f66190b764a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:41:37.428424 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.428397    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175df11e-738d-491f-b1b7-5f66190b764a-kube-api-access-fx5c5" (OuterVolumeSpecName: "kube-api-access-fx5c5") pod "175df11e-738d-491f-b1b7-5f66190b764a" (UID: "175df11e-738d-491f-b1b7-5f66190b764a"). InnerVolumeSpecName "kube-api-access-fx5c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:41:37.527113 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.527033    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx5c5\" (UniqueName: \"kubernetes.io/projected/175df11e-738d-491f-b1b7-5f66190b764a-kube-api-access-fx5c5\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:41:37.527113 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.527064    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175df11e-738d-491f-b1b7-5f66190b764a-proxy-tls\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:41:37.527113 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.527073    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/175df11e-738d-491f-b1b7-5f66190b764a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:41:37.598646 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.598612    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"]
Sep 05 02:41:37.604700 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:37.604675    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-12d88-predictor-55c899946f-kjljc"]
Sep 05 02:41:39.048810 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:39.048773    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="175df11e-738d-491f-b1b7-5f66190b764a" path="/var/lib/kubelet/pods/175df11e-738d-491f-b1b7-5f66190b764a/volumes"
Sep 05 02:41:42.197430 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:42.197345    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j" podUID="cb9f34f3-00f7-483a-87e1-7c8021003116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 05 02:41:42.280046 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:42.280017    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"
Sep 05 02:41:42.280454 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:42.280431    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd" podUID="06f8f0a4-d5d8-47ba-8716-67fe9f7d5597" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 05 02:41:52.196653 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:52.196611    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j" podUID="cb9f34f3-00f7-483a-87e1-7c8021003116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 05 02:41:52.280577 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:41:52.280538    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd" podUID="06f8f0a4-d5d8-47ba-8716-67fe9f7d5597" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 05 02:42:02.198094 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:02.198066    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"
Sep 05 02:42:02.281032 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:02.280996    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd" podUID="06f8f0a4-d5d8-47ba-8716-67fe9f7d5597" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 05 02:42:12.281148 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:12.281108    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd" podUID="06f8f0a4-d5d8-47ba-8716-67fe9f7d5597" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 05 02:42:13.688195 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:13.688162    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:42:14.761125 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:14.761091    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:42:22.280948 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:22.280917    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"
Sep 05 02:42:23.793364 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:23.793338    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:42:23.902539 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:23.902494    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:42:23.915919 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:23.915889    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"]
Sep 05 02:42:23.916170 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:23.916145    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j" podUID="cb9f34f3-00f7-483a-87e1-7c8021003116" containerName="kube-rbac-proxy" containerID="cri-o://c90c43ec0f3c554f1a0766ea7e2b546a76e6dec81abbc80381b06410b7432498" gracePeriod=30
Sep 05 02:42:23.916238 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:23.916136    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j" podUID="cb9f34f3-00f7-483a-87e1-7c8021003116" containerName="kserve-container" containerID="cri-o://1acfde1dfca685c284bea2b05c157dd35c0d291b1b2af49b205699b55fdacd81" gracePeriod=30
Sep 05 02:42:24.007593 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.007561    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:42:24.102267 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.102191    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:42:24.127132 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.127099    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"]
Sep 05 02:42:24.127431 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.127411    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="175df11e-738d-491f-b1b7-5f66190b764a" containerName="kserve-container"
Sep 05 02:42:24.127431 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.127428    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="175df11e-738d-491f-b1b7-5f66190b764a" containerName="kserve-container"
Sep 05 02:42:24.127625 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.127459    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="175df11e-738d-491f-b1b7-5f66190b764a" containerName="kube-rbac-proxy"
Sep 05 02:42:24.127625 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.127468    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="175df11e-738d-491f-b1b7-5f66190b764a" containerName="kube-rbac-proxy"
Sep 05 02:42:24.127625 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.127537    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="175df11e-738d-491f-b1b7-5f66190b764a" containerName="kube-rbac-proxy"
Sep 05 02:42:24.127625 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.127549    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="175df11e-738d-491f-b1b7-5f66190b764a" containerName="kserve-container"
Sep 05 02:42:24.130432 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.130414    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"
Sep 05 02:42:24.132578 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.132557    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-43d66-predictor-serving-cert\""
Sep 05 02:42:24.132854 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.132821    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-43d66-kube-rbac-proxy-sar-config\""
Sep 05 02:42:24.144181 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.144155    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"]
Sep 05 02:42:24.173850 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.173806    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:42:24.271734 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.271696    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cqbd\" (UniqueName: \"kubernetes.io/projected/3bbfbf95-48de-49db-b47c-cdf940039b05-kube-api-access-2cqbd\") pod \"error-404-isvc-43d66-predictor-5cccddb6f-zzgq4\" (UID: \"3bbfbf95-48de-49db-b47c-cdf940039b05\") " pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"
Sep 05 02:42:24.271876 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.271761    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bbfbf95-48de-49db-b47c-cdf940039b05-proxy-tls\") pod \"error-404-isvc-43d66-predictor-5cccddb6f-zzgq4\" (UID: \"3bbfbf95-48de-49db-b47c-cdf940039b05\") " pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"
Sep 05 02:42:24.271876 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.271792    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bbfbf95-48de-49db-b47c-cdf940039b05-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-43d66-predictor-5cccddb6f-zzgq4\" (UID: \"3bbfbf95-48de-49db-b47c-cdf940039b05\") " pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"
Sep 05 02:42:24.372196 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.372163    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bbfbf95-48de-49db-b47c-cdf940039b05-proxy-tls\") pod \"error-404-isvc-43d66-predictor-5cccddb6f-zzgq4\" (UID: \"3bbfbf95-48de-49db-b47c-cdf940039b05\") " pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"
Sep 05 02:42:24.372348 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.372210    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bbfbf95-48de-49db-b47c-cdf940039b05-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-43d66-predictor-5cccddb6f-zzgq4\" (UID: \"3bbfbf95-48de-49db-b47c-cdf940039b05\") " pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"
Sep 05 02:42:24.372348 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.372274    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cqbd\" (UniqueName: \"kubernetes.io/projected/3bbfbf95-48de-49db-b47c-cdf940039b05-kube-api-access-2cqbd\") pod \"error-404-isvc-43d66-predictor-5cccddb6f-zzgq4\" (UID: \"3bbfbf95-48de-49db-b47c-cdf940039b05\") " pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"
Sep 05 02:42:24.372936 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.372913    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bbfbf95-48de-49db-b47c-cdf940039b05-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-43d66-predictor-5cccddb6f-zzgq4\" (UID: \"3bbfbf95-48de-49db-b47c-cdf940039b05\") " pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"
Sep 05 02:42:24.374556 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.374533    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bbfbf95-48de-49db-b47c-cdf940039b05-proxy-tls\") pod \"error-404-isvc-43d66-predictor-5cccddb6f-zzgq4\" (UID: \"3bbfbf95-48de-49db-b47c-cdf940039b05\") " pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"
Sep 05 02:42:24.381236 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.381220    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cqbd\" (UniqueName: \"kubernetes.io/projected/3bbfbf95-48de-49db-b47c-cdf940039b05-kube-api-access-2cqbd\") pod \"error-404-isvc-43d66-predictor-5cccddb6f-zzgq4\" (UID: \"3bbfbf95-48de-49db-b47c-cdf940039b05\") " pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"
Sep 05 02:42:24.401542 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.401511    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:42:24.443586 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.443561    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"
Sep 05 02:42:24.459062 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.459036    2566 generic.go:356] "Generic (PLEG): container finished" podID="cb9f34f3-00f7-483a-87e1-7c8021003116" containerID="c90c43ec0f3c554f1a0766ea7e2b546a76e6dec81abbc80381b06410b7432498" exitCode=2
Sep 05 02:42:24.459170 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.459111    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j" event={"ID":"cb9f34f3-00f7-483a-87e1-7c8021003116","Type":"ContainerDied","Data":"c90c43ec0f3c554f1a0766ea7e2b546a76e6dec81abbc80381b06410b7432498"}
Sep 05 02:42:24.715708 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.715685    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"]
Sep 05 02:42:24.718911 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:42:24.718882    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bbfbf95_48de_49db_b47c_cdf940039b05.slice/crio-d03901f7895af62a9036877b7bae274babb27dc3bd576ad5f8f4375325403bcd WatchSource:0}: Error finding container d03901f7895af62a9036877b7bae274babb27dc3bd576ad5f8f4375325403bcd: Status 404 returned error can't find the container with id d03901f7895af62a9036877b7bae274babb27dc3bd576ad5f8f4375325403bcd
Sep 05 02:42:24.971797 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:24.971707    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:42:25.464763 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:25.464723    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4" event={"ID":"3bbfbf95-48de-49db-b47c-cdf940039b05","Type":"ContainerStarted","Data":"2ee9fbe2bbcae8accb56e6e6eafc6e575a2cf5c662d75fd5414558d0284c83b8"}
Sep 05 02:42:25.464964 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:25.464801    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"
Sep 05 02:42:25.464964 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:25.464937    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"
Sep 05 02:42:25.465043 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:25.464963    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4" event={"ID":"3bbfbf95-48de-49db-b47c-cdf940039b05","Type":"ContainerStarted","Data":"94e3a07784d8f0a988b0d6744f04e9db535735cc7db2932dba994ca391f4695e"}
Sep 05 02:42:25.465043 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:25.464985    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4" event={"ID":"3bbfbf95-48de-49db-b47c-cdf940039b05","Type":"ContainerStarted","Data":"d03901f7895af62a9036877b7bae274babb27dc3bd576ad5f8f4375325403bcd"}
Sep 05 02:42:25.466252 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:25.466227    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4" podUID="3bbfbf95-48de-49db-b47c-cdf940039b05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 05 02:42:25.483777 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:25.483731    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4" podStartSLOduration=1.4837142939999999 podStartE2EDuration="1.483714294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:42:25.482823585 +0000 UTC m=+1476.962217744" watchObservedRunningTime="2026-09-05 02:42:25.483714294 +0000 UTC m=+1476.963108452"
Sep 05 02:42:26.469139 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:26.469094    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4" podUID="3bbfbf95-48de-49db-b47c-cdf940039b05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 05 02:42:27.060114 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:27.060089    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"
Sep 05 02:42:27.194084 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:27.193994    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk4wf\" (UniqueName: \"kubernetes.io/projected/cb9f34f3-00f7-483a-87e1-7c8021003116-kube-api-access-tk4wf\") pod \"cb9f34f3-00f7-483a-87e1-7c8021003116\" (UID: \"cb9f34f3-00f7-483a-87e1-7c8021003116\") "
Sep 05 02:42:27.194084 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:27.194048    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb9f34f3-00f7-483a-87e1-7c8021003116-proxy-tls\") pod \"cb9f34f3-00f7-483a-87e1-7c8021003116\" (UID: \"cb9f34f3-00f7-483a-87e1-7c8021003116\") "
Sep 05 02:42:27.194300 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:27.194113    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb9f34f3-00f7-483a-87e1-7c8021003116-kube-rbac-proxy-sar-config\") pod \"cb9f34f3-00f7-483a-87e1-7c8021003116\" (UID: \"cb9f34f3-00f7-483a-87e1-7c8021003116\") "
Sep 05 02:42:27.194539 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:27.194503    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb9f34f3-00f7-483a-87e1-7c8021003116-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cb9f34f3-00f7-483a-87e1-7c8021003116" (UID: "cb9f34f3-00f7-483a-87e1-7c8021003116"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:42:27.196119 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:27.196099    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb9f34f3-00f7-483a-87e1-7c8021003116-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cb9f34f3-00f7-483a-87e1-7c8021003116" (UID: "cb9f34f3-00f7-483a-87e1-7c8021003116"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:42:27.196211 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:27.196189    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb9f34f3-00f7-483a-87e1-7c8021003116-kube-api-access-tk4wf" (OuterVolumeSpecName: "kube-api-access-tk4wf") pod "cb9f34f3-00f7-483a-87e1-7c8021003116" (UID: "cb9f34f3-00f7-483a-87e1-7c8021003116"). InnerVolumeSpecName "kube-api-access-tk4wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:42:27.295291 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:27.295261    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb9f34f3-00f7-483a-87e1-7c8021003116-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:42:27.295291 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:27.295289    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk4wf\" (UniqueName: \"kubernetes.io/projected/cb9f34f3-00f7-483a-87e1-7c8021003116-kube-api-access-tk4wf\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:42:27.295480 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:27.295302    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb9f34f3-00f7-483a-87e1-7c8021003116-proxy-tls\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:42:27.474111 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:27.474027    2566 generic.go:356] "Generic (PLEG): container finished" podID="cb9f34f3-00f7-483a-87e1-7c8021003116" containerID="1acfde1dfca685c284bea2b05c157dd35c0d291b1b2af49b205699b55fdacd81" exitCode=0
Sep 05 02:42:27.474111 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:27.474077    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"
Sep 05 02:42:27.474111 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:27.474089    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j" event={"ID":"cb9f34f3-00f7-483a-87e1-7c8021003116","Type":"ContainerDied","Data":"1acfde1dfca685c284bea2b05c157dd35c0d291b1b2af49b205699b55fdacd81"}
Sep 05 02:42:27.474654 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:27.474138    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j" event={"ID":"cb9f34f3-00f7-483a-87e1-7c8021003116","Type":"ContainerDied","Data":"80c2811c4718c98cb1bc4a6efbd296100e2b73bc6460b73be37d009040cac15f"}
Sep 05 02:42:27.474654 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:27.474164    2566 scope.go:117] "RemoveContainer" containerID="c90c43ec0f3c554f1a0766ea7e2b546a76e6dec81abbc80381b06410b7432498"
Sep 05 02:42:27.482380 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:27.482355    2566 scope.go:117] "RemoveContainer" containerID="1acfde1dfca685c284bea2b05c157dd35c0d291b1b2af49b205699b55fdacd81"
Sep 05 02:42:27.489575 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:27.489552    2566 scope.go:117] "RemoveContainer" containerID="c90c43ec0f3c554f1a0766ea7e2b546a76e6dec81abbc80381b06410b7432498"
Sep 05 02:42:27.489888 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:42:27.489867    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c90c43ec0f3c554f1a0766ea7e2b546a76e6dec81abbc80381b06410b7432498\": container with ID starting with c90c43ec0f3c554f1a0766ea7e2b546a76e6dec81abbc80381b06410b7432498 not found: ID does not exist" containerID="c90c43ec0f3c554f1a0766ea7e2b546a76e6dec81abbc80381b06410b7432498"
Sep 05 02:42:27.489975 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:27.489901    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c90c43ec0f3c554f1a0766ea7e2b546a76e6dec81abbc80381b06410b7432498"} err="failed to get container status \"c90c43ec0f3c554f1a0766ea7e2b546a76e6dec81abbc80381b06410b7432498\": rpc error: code = NotFound desc = could not find container \"c90c43ec0f3c554f1a0766ea7e2b546a76e6dec81abbc80381b06410b7432498\": container with ID starting with c90c43ec0f3c554f1a0766ea7e2b546a76e6dec81abbc80381b06410b7432498 not found: ID does not exist"
Sep 05 02:42:27.489975 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:27.489924    2566 scope.go:117] "RemoveContainer" containerID="1acfde1dfca685c284bea2b05c157dd35c0d291b1b2af49b205699b55fdacd81"
Sep 05 02:42:27.490219 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:42:27.490194    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1acfde1dfca685c284bea2b05c157dd35c0d291b1b2af49b205699b55fdacd81\": container with ID starting with 1acfde1dfca685c284bea2b05c157dd35c0d291b1b2af49b205699b55fdacd81 not found: ID does not exist" containerID="1acfde1dfca685c284bea2b05c157dd35c0d291b1b2af49b205699b55fdacd81"
Sep 05 02:42:27.490263 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:27.490228    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1acfde1dfca685c284bea2b05c157dd35c0d291b1b2af49b205699b55fdacd81"} err="failed to get container status \"1acfde1dfca685c284bea2b05c157dd35c0d291b1b2af49b205699b55fdacd81\": rpc error: code = NotFound desc = could not find container \"1acfde1dfca685c284bea2b05c157dd35c0d291b1b2af49b205699b55fdacd81\": container with ID starting with 1acfde1dfca685c284bea2b05c157dd35c0d291b1b2af49b205699b55fdacd81 not found: ID does not exist"
Sep 05 02:42:27.494735 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:27.494713    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"]
Sep 05 02:42:27.498162 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:27.498141    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-80c39-predictor-58b9fd5fdf-kf94j"]
Sep 05 02:42:29.047523 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:29.047492    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb9f34f3-00f7-483a-87e1-7c8021003116" path="/var/lib/kubelet/pods/cb9f34f3-00f7-483a-87e1-7c8021003116/volumes"
Sep 05 02:42:31.473372 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:31.473342    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"
Sep 05 02:42:31.473746 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:31.473721    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4" podUID="3bbfbf95-48de-49db-b47c-cdf940039b05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 05 02:42:34.123108 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:34.123074    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:42:34.259718 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:34.259686    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:42:41.474151 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:41.474110    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4" podUID="3bbfbf95-48de-49db-b47c-cdf940039b05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 05 02:42:51.474185 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:42:51.474141    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4" podUID="3bbfbf95-48de-49db-b47c-cdf940039b05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 05 02:43:01.474061 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:43:01.474020    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4" podUID="3bbfbf95-48de-49db-b47c-cdf940039b05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 05 02:43:11.475072 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:43:11.474993    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"
Sep 05 02:43:35.656447 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:43:35.656412    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:43:35.712732 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:43:35.712705    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:43:56.102720 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:43:56.102684    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:44:07.990776 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:44:07.990742    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:47:38.868946 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:47:38.868911    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:03.428732 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:03.428699    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:03.480190 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:03.480161    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:48.864753 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:48.864679    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:49.014811 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.014779    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:49.090553 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.090520    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:49.163184 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.163149    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"]
Sep 05 02:50:49.163458 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.163436    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:49.163555 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.163542    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb9f34f3-00f7-483a-87e1-7c8021003116" containerName="kserve-container"
Sep 05 02:50:49.163594 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.163559    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb9f34f3-00f7-483a-87e1-7c8021003116" containerName="kserve-container"
Sep 05 02:50:49.163627 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.163596    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb9f34f3-00f7-483a-87e1-7c8021003116" containerName="kube-rbac-proxy"
Sep 05 02:50:49.163627 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.163604    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb9f34f3-00f7-483a-87e1-7c8021003116" containerName="kube-rbac-proxy"
Sep 05 02:50:49.163700 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.163690    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb9f34f3-00f7-483a-87e1-7c8021003116" containerName="kube-rbac-proxy"
Sep 05 02:50:49.163734 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.163704    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb9f34f3-00f7-483a-87e1-7c8021003116" containerName="kserve-container"
Sep 05 02:50:49.165805 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.165780    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"
Sep 05 02:50:49.168430 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.168411    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ca9bf-predictor-serving-cert\""
Sep 05 02:50:49.168522 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.168419    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ca9bf-kube-rbac-proxy-sar-config\""
Sep 05 02:50:49.178487 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.178121    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"]
Sep 05 02:50:49.203548 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.203525    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"]
Sep 05 02:50:49.203796 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.203771    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd" podUID="06f8f0a4-d5d8-47ba-8716-67fe9f7d5597" containerName="kserve-container" containerID="cri-o://00143145a00e20a6627de96712c5dcf57dae0b2e769e2874576a504effed70d9" gracePeriod=30
Sep 05 02:50:49.203898 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.203801    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd" podUID="06f8f0a4-d5d8-47ba-8716-67fe9f7d5597" containerName="kube-rbac-proxy" containerID="cri-o://6ffaf25c8a20dab52293bf59f8e6d6d8a4f90bda52bfe02419cfc00a32cb5272" gracePeriod=30
Sep 05 02:50:49.233823 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.233795    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:49.292291 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.292260    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd23f079-4a3b-4bbd-a05a-d5e22030d087-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll\" (UID: \"dd23f079-4a3b-4bbd-a05a-d5e22030d087\") " pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"
Sep 05 02:50:49.292491 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.292297    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5vtz\" (UniqueName: \"kubernetes.io/projected/dd23f079-4a3b-4bbd-a05a-d5e22030d087-kube-api-access-n5vtz\") pod \"error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll\" (UID: \"dd23f079-4a3b-4bbd-a05a-d5e22030d087\") " pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"
Sep 05 02:50:49.292491 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.292338    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd23f079-4a3b-4bbd-a05a-d5e22030d087-proxy-tls\") pod \"error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll\" (UID: \"dd23f079-4a3b-4bbd-a05a-d5e22030d087\") " pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"
Sep 05 02:50:49.365716 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.365685    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:49.393527 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.393498    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd23f079-4a3b-4bbd-a05a-d5e22030d087-proxy-tls\") pod \"error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll\" (UID: \"dd23f079-4a3b-4bbd-a05a-d5e22030d087\") " pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"
Sep 05 02:50:49.393688 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.393578    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd23f079-4a3b-4bbd-a05a-d5e22030d087-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll\" (UID: \"dd23f079-4a3b-4bbd-a05a-d5e22030d087\") " pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"
Sep 05 02:50:49.393688 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.393612    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5vtz\" (UniqueName: \"kubernetes.io/projected/dd23f079-4a3b-4bbd-a05a-d5e22030d087-kube-api-access-n5vtz\") pod \"error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll\" (UID: \"dd23f079-4a3b-4bbd-a05a-d5e22030d087\") " pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"
Sep 05 02:50:49.393688 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:50:49.393618    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-serving-cert: secret "error-404-isvc-ca9bf-predictor-serving-cert" not found
Sep 05 02:50:49.393688 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:50:49.393678    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd23f079-4a3b-4bbd-a05a-d5e22030d087-proxy-tls podName:dd23f079-4a3b-4bbd-a05a-d5e22030d087 nodeName:}" failed. No retries permitted until 2026-09-05 02:50:49.893660939 +0000 UTC m=+1981.373055075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dd23f079-4a3b-4bbd-a05a-d5e22030d087-proxy-tls") pod "error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll" (UID: "dd23f079-4a3b-4bbd-a05a-d5e22030d087") : secret "error-404-isvc-ca9bf-predictor-serving-cert" not found
Sep 05 02:50:49.394276 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.394255    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd23f079-4a3b-4bbd-a05a-d5e22030d087-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll\" (UID: \"dd23f079-4a3b-4bbd-a05a-d5e22030d087\") " pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"
Sep 05 02:50:49.402329 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.402303    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5vtz\" (UniqueName: \"kubernetes.io/projected/dd23f079-4a3b-4bbd-a05a-d5e22030d087-kube-api-access-n5vtz\") pod \"error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll\" (UID: \"dd23f079-4a3b-4bbd-a05a-d5e22030d087\") " pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"
Sep 05 02:50:49.662985 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.662953    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:49.897690 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.897657    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd23f079-4a3b-4bbd-a05a-d5e22030d087-proxy-tls\") pod \"error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll\" (UID: \"dd23f079-4a3b-4bbd-a05a-d5e22030d087\") " pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"
Sep 05 02:50:49.900062 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:49.900035    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd23f079-4a3b-4bbd-a05a-d5e22030d087-proxy-tls\") pod \"error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll\" (UID: \"dd23f079-4a3b-4bbd-a05a-d5e22030d087\") " pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"
Sep 05 02:50:50.077060 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:50.076966    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"
Sep 05 02:50:50.291742 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:50.291704    2566 generic.go:356] "Generic (PLEG): container finished" podID="06f8f0a4-d5d8-47ba-8716-67fe9f7d5597" containerID="6ffaf25c8a20dab52293bf59f8e6d6d8a4f90bda52bfe02419cfc00a32cb5272" exitCode=2
Sep 05 02:50:50.291924 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:50.291751    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd" event={"ID":"06f8f0a4-d5d8-47ba-8716-67fe9f7d5597","Type":"ContainerDied","Data":"6ffaf25c8a20dab52293bf59f8e6d6d8a4f90bda52bfe02419cfc00a32cb5272"}
Sep 05 02:50:50.351417 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:50.351321    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"]
Sep 05 02:50:50.356195 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:50:50.356150    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd23f079_4a3b_4bbd_a05a_d5e22030d087.slice/crio-da2146835457c01ff21ca40e718d1bfda3a1c522ab199c7b8f0bfae8d35a5b01 WatchSource:0}: Error finding container da2146835457c01ff21ca40e718d1bfda3a1c522ab199c7b8f0bfae8d35a5b01: Status 404 returned error can't find the container with id da2146835457c01ff21ca40e718d1bfda3a1c522ab199c7b8f0bfae8d35a5b01
Sep 05 02:50:51.297067 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:51.297034    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll" event={"ID":"dd23f079-4a3b-4bbd-a05a-d5e22030d087","Type":"ContainerStarted","Data":"ddfe7c885ce71c0bd84919331ff3326d3840530e1d8acc62ef1eb9351aebea82"}
Sep 05 02:50:51.297468 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:51.297088    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll" event={"ID":"dd23f079-4a3b-4bbd-a05a-d5e22030d087","Type":"ContainerStarted","Data":"1cc1eaea7cb53c4e68a674d24a7b5e22363b23176cb98c25960f01018de1ff97"}
Sep 05 02:50:51.297468 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:51.297109    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll" event={"ID":"dd23f079-4a3b-4bbd-a05a-d5e22030d087","Type":"ContainerStarted","Data":"da2146835457c01ff21ca40e718d1bfda3a1c522ab199c7b8f0bfae8d35a5b01"}
Sep 05 02:50:51.297468 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:51.297181    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"
Sep 05 02:50:51.314483 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:51.314420    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll" podStartSLOduration=2.314401353 podStartE2EDuration="2.314401353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:50:51.314169493 +0000 UTC m=+1982.793563662" watchObservedRunningTime="2026-09-05 02:50:51.314401353 +0000 UTC m=+1982.793795516"
Sep 05 02:50:52.276263 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:52.276219    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd" podUID="06f8f0a4-d5d8-47ba-8716-67fe9f7d5597" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 05 02:50:52.280673 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:52.280645    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd" podUID="06f8f0a4-d5d8-47ba-8716-67fe9f7d5597" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 05 02:50:52.300320 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:52.300291    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"
Sep 05 02:50:52.301636 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:52.301610    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll" podUID="dd23f079-4a3b-4bbd-a05a-d5e22030d087" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 05 02:50:52.438269 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:52.438239    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"
Sep 05 02:50:52.526863 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:52.526762    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsdsc\" (UniqueName: \"kubernetes.io/projected/06f8f0a4-d5d8-47ba-8716-67fe9f7d5597-kube-api-access-xsdsc\") pod \"06f8f0a4-d5d8-47ba-8716-67fe9f7d5597\" (UID: \"06f8f0a4-d5d8-47ba-8716-67fe9f7d5597\") "
Sep 05 02:50:52.526863 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:52.526806    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06f8f0a4-d5d8-47ba-8716-67fe9f7d5597-proxy-tls\") pod \"06f8f0a4-d5d8-47ba-8716-67fe9f7d5597\" (UID: \"06f8f0a4-d5d8-47ba-8716-67fe9f7d5597\") "
Sep 05 02:50:52.527042 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:52.526871    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06f8f0a4-d5d8-47ba-8716-67fe9f7d5597-kube-rbac-proxy-sar-config\") pod \"06f8f0a4-d5d8-47ba-8716-67fe9f7d5597\" (UID: \"06f8f0a4-d5d8-47ba-8716-67fe9f7d5597\") "
Sep 05 02:50:52.527254 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:52.527220    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f8f0a4-d5d8-47ba-8716-67fe9f7d5597-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "06f8f0a4-d5d8-47ba-8716-67fe9f7d5597" (UID: "06f8f0a4-d5d8-47ba-8716-67fe9f7d5597"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:50:52.528965 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:52.528934    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06f8f0a4-d5d8-47ba-8716-67fe9f7d5597-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "06f8f0a4-d5d8-47ba-8716-67fe9f7d5597" (UID: "06f8f0a4-d5d8-47ba-8716-67fe9f7d5597"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:50:52.529051 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:52.528942    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f8f0a4-d5d8-47ba-8716-67fe9f7d5597-kube-api-access-xsdsc" (OuterVolumeSpecName: "kube-api-access-xsdsc") pod "06f8f0a4-d5d8-47ba-8716-67fe9f7d5597" (UID: "06f8f0a4-d5d8-47ba-8716-67fe9f7d5597"). InnerVolumeSpecName "kube-api-access-xsdsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:50:52.628082 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:52.628037    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsdsc\" (UniqueName: \"kubernetes.io/projected/06f8f0a4-d5d8-47ba-8716-67fe9f7d5597-kube-api-access-xsdsc\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:50:52.628082 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:52.628074    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06f8f0a4-d5d8-47ba-8716-67fe9f7d5597-proxy-tls\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:50:52.628082 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:52.628084    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06f8f0a4-d5d8-47ba-8716-67fe9f7d5597-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:50:53.305019 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:53.304930    2566 generic.go:356] "Generic (PLEG): container finished" podID="06f8f0a4-d5d8-47ba-8716-67fe9f7d5597" containerID="00143145a00e20a6627de96712c5dcf57dae0b2e769e2874576a504effed70d9" exitCode=0
Sep 05 02:50:53.305019 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:53.304984    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"
Sep 05 02:50:53.305528 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:53.305022    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd" event={"ID":"06f8f0a4-d5d8-47ba-8716-67fe9f7d5597","Type":"ContainerDied","Data":"00143145a00e20a6627de96712c5dcf57dae0b2e769e2874576a504effed70d9"}
Sep 05 02:50:53.305528 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:53.305094    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd" event={"ID":"06f8f0a4-d5d8-47ba-8716-67fe9f7d5597","Type":"ContainerDied","Data":"ffdb61fce00ca1d925197c189617b310160fcec4e7f28df8f1b2183dd05aff96"}
Sep 05 02:50:53.305528 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:53.305136    2566 scope.go:117] "RemoveContainer" containerID="6ffaf25c8a20dab52293bf59f8e6d6d8a4f90bda52bfe02419cfc00a32cb5272"
Sep 05 02:50:53.305679 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:53.305635    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll" podUID="dd23f079-4a3b-4bbd-a05a-d5e22030d087" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 05 02:50:53.313569 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:53.313554    2566 scope.go:117] "RemoveContainer" containerID="00143145a00e20a6627de96712c5dcf57dae0b2e769e2874576a504effed70d9"
Sep 05 02:50:53.320683 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:53.320660    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"]
Sep 05 02:50:53.321490 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:53.321399    2566 scope.go:117] "RemoveContainer" containerID="6ffaf25c8a20dab52293bf59f8e6d6d8a4f90bda52bfe02419cfc00a32cb5272"
Sep 05 02:50:53.321695 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:50:53.321679    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ffaf25c8a20dab52293bf59f8e6d6d8a4f90bda52bfe02419cfc00a32cb5272\": container with ID starting with 6ffaf25c8a20dab52293bf59f8e6d6d8a4f90bda52bfe02419cfc00a32cb5272 not found: ID does not exist" containerID="6ffaf25c8a20dab52293bf59f8e6d6d8a4f90bda52bfe02419cfc00a32cb5272"
Sep 05 02:50:53.321734 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:53.321704    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ffaf25c8a20dab52293bf59f8e6d6d8a4f90bda52bfe02419cfc00a32cb5272"} err="failed to get container status \"6ffaf25c8a20dab52293bf59f8e6d6d8a4f90bda52bfe02419cfc00a32cb5272\": rpc error: code = NotFound desc = could not find container \"6ffaf25c8a20dab52293bf59f8e6d6d8a4f90bda52bfe02419cfc00a32cb5272\": container with ID starting with 6ffaf25c8a20dab52293bf59f8e6d6d8a4f90bda52bfe02419cfc00a32cb5272 not found: ID does not exist"
Sep 05 02:50:53.321734 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:53.321721    2566 scope.go:117] "RemoveContainer" containerID="00143145a00e20a6627de96712c5dcf57dae0b2e769e2874576a504effed70d9"
Sep 05 02:50:53.321970 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:50:53.321954    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00143145a00e20a6627de96712c5dcf57dae0b2e769e2874576a504effed70d9\": container with ID starting with 00143145a00e20a6627de96712c5dcf57dae0b2e769e2874576a504effed70d9 not found: ID does not exist" containerID="00143145a00e20a6627de96712c5dcf57dae0b2e769e2874576a504effed70d9"
Sep 05 02:50:53.322009 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:53.321976    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00143145a00e20a6627de96712c5dcf57dae0b2e769e2874576a504effed70d9"} err="failed to get container status \"00143145a00e20a6627de96712c5dcf57dae0b2e769e2874576a504effed70d9\": rpc error: code = NotFound desc = could not find container \"00143145a00e20a6627de96712c5dcf57dae0b2e769e2874576a504effed70d9\": container with ID starting with 00143145a00e20a6627de96712c5dcf57dae0b2e769e2874576a504effed70d9 not found: ID does not exist"
Sep 05 02:50:53.325772 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:53.325753    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9755b-predictor-58df99b7-jlvhd"]
Sep 05 02:50:55.048345 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:55.048309    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06f8f0a4-d5d8-47ba-8716-67fe9f7d5597" path="/var/lib/kubelet/pods/06f8f0a4-d5d8-47ba-8716-67fe9f7d5597/volumes"
Sep 05 02:50:58.310771 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:58.310740    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"
Sep 05 02:50:58.311263 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:50:58.311232    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll" podUID="dd23f079-4a3b-4bbd-a05a-d5e22030d087" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 05 02:51:08.311893 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:08.311850    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll" podUID="dd23f079-4a3b-4bbd-a05a-d5e22030d087" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 05 02:51:18.311510 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:18.311467    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll" podUID="dd23f079-4a3b-4bbd-a05a-d5e22030d087" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 05 02:51:28.311181 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:28.311142    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll" podUID="dd23f079-4a3b-4bbd-a05a-d5e22030d087" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 05 02:51:38.312003 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:38.311970    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"
Sep 05 02:51:48.874137 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:48.874096    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"]
Sep 05 02:51:48.874625 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:48.874316    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4" podUID="3bbfbf95-48de-49db-b47c-cdf940039b05" containerName="kserve-container" containerID="cri-o://94e3a07784d8f0a988b0d6744f04e9db535735cc7db2932dba994ca391f4695e" gracePeriod=30
Sep 05 02:51:48.874625 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:48.874366    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4" podUID="3bbfbf95-48de-49db-b47c-cdf940039b05" containerName="kube-rbac-proxy" containerID="cri-o://2ee9fbe2bbcae8accb56e6e6eafc6e575a2cf5c662d75fd5414558d0284c83b8" gracePeriod=30
Sep 05 02:51:49.114439 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.114402    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"]
Sep 05 02:51:49.114863 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.114827    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06f8f0a4-d5d8-47ba-8716-67fe9f7d5597" containerName="kserve-container"
Sep 05 02:51:49.115013 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.115000    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="06f8f0a4-d5d8-47ba-8716-67fe9f7d5597" containerName="kserve-container"
Sep 05 02:51:49.115141 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.115128    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06f8f0a4-d5d8-47ba-8716-67fe9f7d5597" containerName="kube-rbac-proxy"
Sep 05 02:51:49.115255 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.115227    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="06f8f0a4-d5d8-47ba-8716-67fe9f7d5597" containerName="kube-rbac-proxy"
Sep 05 02:51:49.115367 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.115333    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="06f8f0a4-d5d8-47ba-8716-67fe9f7d5597" containerName="kube-rbac-proxy"
Sep 05 02:51:49.115367 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.115347    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="06f8f0a4-d5d8-47ba-8716-67fe9f7d5597" containerName="kserve-container"
Sep 05 02:51:49.118689 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.118672    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"
Sep 05 02:51:49.121046 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.121023    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3af6d-predictor-serving-cert\""
Sep 05 02:51:49.121143 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.121051    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3af6d-kube-rbac-proxy-sar-config\""
Sep 05 02:51:49.128029 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.127993    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"]
Sep 05 02:51:49.233234 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.233199    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae4118a5-a694-49d0-92d9-d4f154d69c57-proxy-tls\") pod \"error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4\" (UID: \"ae4118a5-a694-49d0-92d9-d4f154d69c57\") " pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"
Sep 05 02:51:49.233234 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.233242    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae4118a5-a694-49d0-92d9-d4f154d69c57-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4\" (UID: \"ae4118a5-a694-49d0-92d9-d4f154d69c57\") " pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"
Sep 05 02:51:49.233472 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.233344    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52ndl\" (UniqueName: \"kubernetes.io/projected/ae4118a5-a694-49d0-92d9-d4f154d69c57-kube-api-access-52ndl\") pod \"error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4\" (UID: \"ae4118a5-a694-49d0-92d9-d4f154d69c57\") " pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"
Sep 05 02:51:49.334361 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.334318    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae4118a5-a694-49d0-92d9-d4f154d69c57-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4\" (UID: \"ae4118a5-a694-49d0-92d9-d4f154d69c57\") " pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"
Sep 05 02:51:49.334615 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.334390    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52ndl\" (UniqueName: \"kubernetes.io/projected/ae4118a5-a694-49d0-92d9-d4f154d69c57-kube-api-access-52ndl\") pod \"error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4\" (UID: \"ae4118a5-a694-49d0-92d9-d4f154d69c57\") " pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"
Sep 05 02:51:49.334615 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.334443    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae4118a5-a694-49d0-92d9-d4f154d69c57-proxy-tls\") pod \"error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4\" (UID: \"ae4118a5-a694-49d0-92d9-d4f154d69c57\") " pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"
Sep 05 02:51:49.334615 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:51:49.334564    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-serving-cert: secret "error-404-isvc-3af6d-predictor-serving-cert" not found
Sep 05 02:51:49.334777 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:51:49.334634    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae4118a5-a694-49d0-92d9-d4f154d69c57-proxy-tls podName:ae4118a5-a694-49d0-92d9-d4f154d69c57 nodeName:}" failed. No retries permitted until 2026-09-05 02:51:49.834613273 +0000 UTC m=+2041.314007416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ae4118a5-a694-49d0-92d9-d4f154d69c57-proxy-tls") pod "error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4" (UID: "ae4118a5-a694-49d0-92d9-d4f154d69c57") : secret "error-404-isvc-3af6d-predictor-serving-cert" not found
Sep 05 02:51:49.335123 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.335099    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae4118a5-a694-49d0-92d9-d4f154d69c57-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4\" (UID: \"ae4118a5-a694-49d0-92d9-d4f154d69c57\") " pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"
Sep 05 02:51:49.344618 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.344591    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52ndl\" (UniqueName: \"kubernetes.io/projected/ae4118a5-a694-49d0-92d9-d4f154d69c57-kube-api-access-52ndl\") pod \"error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4\" (UID: \"ae4118a5-a694-49d0-92d9-d4f154d69c57\") " pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"
Sep 05 02:51:49.506966 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.506880    2566 generic.go:356] "Generic (PLEG): container finished" podID="3bbfbf95-48de-49db-b47c-cdf940039b05" containerID="2ee9fbe2bbcae8accb56e6e6eafc6e575a2cf5c662d75fd5414558d0284c83b8" exitCode=2
Sep 05 02:51:49.506966 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.506926    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4" event={"ID":"3bbfbf95-48de-49db-b47c-cdf940039b05","Type":"ContainerDied","Data":"2ee9fbe2bbcae8accb56e6e6eafc6e575a2cf5c662d75fd5414558d0284c83b8"}
Sep 05 02:51:49.535899 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.535866    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:49.589054 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.589024    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:49.838930 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.838825    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae4118a5-a694-49d0-92d9-d4f154d69c57-proxy-tls\") pod \"error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4\" (UID: \"ae4118a5-a694-49d0-92d9-d4f154d69c57\") " pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"
Sep 05 02:51:49.841258 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:49.841240    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae4118a5-a694-49d0-92d9-d4f154d69c57-proxy-tls\") pod \"error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4\" (UID: \"ae4118a5-a694-49d0-92d9-d4f154d69c57\") " pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"
Sep 05 02:51:50.031686 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:50.031656    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"
Sep 05 02:51:50.301037 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:50.301011    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"]
Sep 05 02:51:50.303449 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:51:50.303423    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4118a5_a694_49d0_92d9_d4f154d69c57.slice/crio-287c95be279ef842618f1e9bb06357be1afc8443614d5c40106c1f553a8a4a07 WatchSource:0}: Error finding container 287c95be279ef842618f1e9bb06357be1afc8443614d5c40106c1f553a8a4a07: Status 404 returned error can't find the container with id 287c95be279ef842618f1e9bb06357be1afc8443614d5c40106c1f553a8a4a07
Sep 05 02:51:50.454678 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:50.454650    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:50.514351 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:50.514315    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4" event={"ID":"ae4118a5-a694-49d0-92d9-d4f154d69c57","Type":"ContainerStarted","Data":"a7edf3822d16c7db7ff5348ceba1e242bb3fd5912caa0c6d8fea37440720f86c"}
Sep 05 02:51:50.514552 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:50.514407    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4" event={"ID":"ae4118a5-a694-49d0-92d9-d4f154d69c57","Type":"ContainerStarted","Data":"7d38953c8e00e0bd7bd53c165870bd7c60bedaec12f04e896fbf261be69c7d6a"}
Sep 05 02:51:50.514552 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:50.514440    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4" event={"ID":"ae4118a5-a694-49d0-92d9-d4f154d69c57","Type":"ContainerStarted","Data":"287c95be279ef842618f1e9bb06357be1afc8443614d5c40106c1f553a8a4a07"}
Sep 05 02:51:50.514552 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:50.514502    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"
Sep 05 02:51:50.530531 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:50.530461    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4" podStartSLOduration=1.530441728 podStartE2EDuration="1.530441728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:51:50.530066373 +0000 UTC m=+2042.009460532" watchObservedRunningTime="2026-09-05 02:51:50.530441728 +0000 UTC m=+2042.009835887"
Sep 05 02:51:51.469380 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:51.469333    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4" podUID="3bbfbf95-48de-49db-b47c-cdf940039b05" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 05 02:51:51.474239 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:51.474213    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4" podUID="3bbfbf95-48de-49db-b47c-cdf940039b05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 05 02:51:51.518668 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:51.518629    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"
Sep 05 02:51:51.519940 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:51.519910    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4" podUID="ae4118a5-a694-49d0-92d9-d4f154d69c57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 05 02:51:52.113114 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.113082    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"
Sep 05 02:51:52.152784 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.152753    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bbfbf95-48de-49db-b47c-cdf940039b05-kube-rbac-proxy-sar-config\") pod \"3bbfbf95-48de-49db-b47c-cdf940039b05\" (UID: \"3bbfbf95-48de-49db-b47c-cdf940039b05\") "
Sep 05 02:51:52.152980 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.152798    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bbfbf95-48de-49db-b47c-cdf940039b05-proxy-tls\") pod \"3bbfbf95-48de-49db-b47c-cdf940039b05\" (UID: \"3bbfbf95-48de-49db-b47c-cdf940039b05\") "
Sep 05 02:51:52.152980 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.152827    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cqbd\" (UniqueName: \"kubernetes.io/projected/3bbfbf95-48de-49db-b47c-cdf940039b05-kube-api-access-2cqbd\") pod \"3bbfbf95-48de-49db-b47c-cdf940039b05\" (UID: \"3bbfbf95-48de-49db-b47c-cdf940039b05\") "
Sep 05 02:51:52.153202 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.153176    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bbfbf95-48de-49db-b47c-cdf940039b05-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3bbfbf95-48de-49db-b47c-cdf940039b05" (UID: "3bbfbf95-48de-49db-b47c-cdf940039b05"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:51:52.154912 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.154886    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bbfbf95-48de-49db-b47c-cdf940039b05-kube-api-access-2cqbd" (OuterVolumeSpecName: "kube-api-access-2cqbd") pod "3bbfbf95-48de-49db-b47c-cdf940039b05" (UID: "3bbfbf95-48de-49db-b47c-cdf940039b05"). InnerVolumeSpecName "kube-api-access-2cqbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:51:52.155014 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.154992    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bbfbf95-48de-49db-b47c-cdf940039b05-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3bbfbf95-48de-49db-b47c-cdf940039b05" (UID: "3bbfbf95-48de-49db-b47c-cdf940039b05"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:51:52.254318 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.254232    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bbfbf95-48de-49db-b47c-cdf940039b05-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:51:52.254318 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.254262    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bbfbf95-48de-49db-b47c-cdf940039b05-proxy-tls\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:51:52.254318 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.254277    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cqbd\" (UniqueName: \"kubernetes.io/projected/3bbfbf95-48de-49db-b47c-cdf940039b05-kube-api-access-2cqbd\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:51:52.524034 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.523935    2566 generic.go:356] "Generic (PLEG): container finished" podID="3bbfbf95-48de-49db-b47c-cdf940039b05" containerID="94e3a07784d8f0a988b0d6744f04e9db535735cc7db2932dba994ca391f4695e" exitCode=0
Sep 05 02:51:52.524034 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.523994    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"
Sep 05 02:51:52.524034 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.524025    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4" event={"ID":"3bbfbf95-48de-49db-b47c-cdf940039b05","Type":"ContainerDied","Data":"94e3a07784d8f0a988b0d6744f04e9db535735cc7db2932dba994ca391f4695e"}
Sep 05 02:51:52.524588 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.524115    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4" event={"ID":"3bbfbf95-48de-49db-b47c-cdf940039b05","Type":"ContainerDied","Data":"d03901f7895af62a9036877b7bae274babb27dc3bd576ad5f8f4375325403bcd"}
Sep 05 02:51:52.524588 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.524143    2566 scope.go:117] "RemoveContainer" containerID="2ee9fbe2bbcae8accb56e6e6eafc6e575a2cf5c662d75fd5414558d0284c83b8"
Sep 05 02:51:52.524760 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.524729    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4" podUID="ae4118a5-a694-49d0-92d9-d4f154d69c57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 05 02:51:52.532047 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.532024    2566 scope.go:117] "RemoveContainer" containerID="94e3a07784d8f0a988b0d6744f04e9db535735cc7db2932dba994ca391f4695e"
Sep 05 02:51:52.538951 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.538935    2566 scope.go:117] "RemoveContainer" containerID="2ee9fbe2bbcae8accb56e6e6eafc6e575a2cf5c662d75fd5414558d0284c83b8"
Sep 05 02:51:52.539196 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:51:52.539168    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ee9fbe2bbcae8accb56e6e6eafc6e575a2cf5c662d75fd5414558d0284c83b8\": container with ID starting with 2ee9fbe2bbcae8accb56e6e6eafc6e575a2cf5c662d75fd5414558d0284c83b8 not found: ID does not exist" containerID="2ee9fbe2bbcae8accb56e6e6eafc6e575a2cf5c662d75fd5414558d0284c83b8"
Sep 05 02:51:52.539258 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.539195    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee9fbe2bbcae8accb56e6e6eafc6e575a2cf5c662d75fd5414558d0284c83b8"} err="failed to get container status \"2ee9fbe2bbcae8accb56e6e6eafc6e575a2cf5c662d75fd5414558d0284c83b8\": rpc error: code = NotFound desc = could not find container \"2ee9fbe2bbcae8accb56e6e6eafc6e575a2cf5c662d75fd5414558d0284c83b8\": container with ID starting with 2ee9fbe2bbcae8accb56e6e6eafc6e575a2cf5c662d75fd5414558d0284c83b8 not found: ID does not exist"
Sep 05 02:51:52.539258 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.539211    2566 scope.go:117] "RemoveContainer" containerID="94e3a07784d8f0a988b0d6744f04e9db535735cc7db2932dba994ca391f4695e"
Sep 05 02:51:52.539435 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:51:52.539418    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94e3a07784d8f0a988b0d6744f04e9db535735cc7db2932dba994ca391f4695e\": container with ID starting with 94e3a07784d8f0a988b0d6744f04e9db535735cc7db2932dba994ca391f4695e not found: ID does not exist" containerID="94e3a07784d8f0a988b0d6744f04e9db535735cc7db2932dba994ca391f4695e"
Sep 05 02:51:52.539478 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.539442    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94e3a07784d8f0a988b0d6744f04e9db535735cc7db2932dba994ca391f4695e"} err="failed to get container status \"94e3a07784d8f0a988b0d6744f04e9db535735cc7db2932dba994ca391f4695e\": rpc error: code = NotFound desc = could not find container \"94e3a07784d8f0a988b0d6744f04e9db535735cc7db2932dba994ca391f4695e\": container with ID starting with 94e3a07784d8f0a988b0d6744f04e9db535735cc7db2932dba994ca391f4695e not found: ID does not exist"
Sep 05 02:51:52.545484 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.545461    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"]
Sep 05 02:51:52.549863 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:52.549843    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43d66-predictor-5cccddb6f-zzgq4"]
Sep 05 02:51:53.047631 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:53.047597    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bbfbf95-48de-49db-b47c-cdf940039b05" path="/var/lib/kubelet/pods/3bbfbf95-48de-49db-b47c-cdf940039b05/volumes"
Sep 05 02:51:57.530036 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:57.530005    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"
Sep 05 02:51:57.530600 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:57.530572    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4" podUID="ae4118a5-a694-49d0-92d9-d4f154d69c57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 05 02:51:59.178335 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.178301    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:59.280756 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.280720    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"]
Sep 05 02:51:59.281070 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.281041    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll" podUID="dd23f079-4a3b-4bbd-a05a-d5e22030d087" containerName="kserve-container" containerID="cri-o://1cc1eaea7cb53c4e68a674d24a7b5e22363b23176cb98c25960f01018de1ff97" gracePeriod=30
Sep 05 02:51:59.281212 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.281178    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll" podUID="dd23f079-4a3b-4bbd-a05a-d5e22030d087" containerName="kube-rbac-proxy" containerID="cri-o://ddfe7c885ce71c0bd84919331ff3326d3840530e1d8acc62ef1eb9351aebea82" gracePeriod=30
Sep 05 02:51:59.483451 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.483362    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"]
Sep 05 02:51:59.483653 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.483640    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bbfbf95-48de-49db-b47c-cdf940039b05" containerName="kserve-container"
Sep 05 02:51:59.483653 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.483654    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bbfbf95-48de-49db-b47c-cdf940039b05" containerName="kserve-container"
Sep 05 02:51:59.483769 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.483668    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bbfbf95-48de-49db-b47c-cdf940039b05" containerName="kube-rbac-proxy"
Sep 05 02:51:59.483769 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.483674    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bbfbf95-48de-49db-b47c-cdf940039b05" containerName="kube-rbac-proxy"
Sep 05 02:51:59.483769 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.483722    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bbfbf95-48de-49db-b47c-cdf940039b05" containerName="kube-rbac-proxy"
Sep 05 02:51:59.483769 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.483730    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bbfbf95-48de-49db-b47c-cdf940039b05" containerName="kserve-container"
Sep 05 02:51:59.486641 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.486622    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"
Sep 05 02:51:59.488822 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.488797    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a4a3e-predictor-serving-cert\""
Sep 05 02:51:59.488953 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.488902    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a4a3e-kube-rbac-proxy-sar-config\""
Sep 05 02:51:59.492724 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.492700    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:59.495987 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.495964    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"]
Sep 05 02:51:59.552737 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.552707    2566 generic.go:356] "Generic (PLEG): container finished" podID="dd23f079-4a3b-4bbd-a05a-d5e22030d087" containerID="ddfe7c885ce71c0bd84919331ff3326d3840530e1d8acc62ef1eb9351aebea82" exitCode=2
Sep 05 02:51:59.552890 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.552785    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll" event={"ID":"dd23f079-4a3b-4bbd-a05a-d5e22030d087","Type":"ContainerDied","Data":"ddfe7c885ce71c0bd84919331ff3326d3840530e1d8acc62ef1eb9351aebea82"}
Sep 05 02:51:59.560172 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.560138    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:59.611446 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.611410    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x989\" (UniqueName: \"kubernetes.io/projected/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-kube-api-access-2x989\") pod \"error-404-isvc-a4a3e-predictor-6667499d67-jztfn\" (UID: \"2fdd3964-fd57-4a4c-8706-7db9f2de6e47\") " pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"
Sep 05 02:51:59.611601 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.611470    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a4a3e-predictor-6667499d67-jztfn\" (UID: \"2fdd3964-fd57-4a4c-8706-7db9f2de6e47\") " pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"
Sep 05 02:51:59.611601 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.611524    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-proxy-tls\") pod \"error-404-isvc-a4a3e-predictor-6667499d67-jztfn\" (UID: \"2fdd3964-fd57-4a4c-8706-7db9f2de6e47\") " pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"
Sep 05 02:51:59.712498 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.712458    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a4a3e-predictor-6667499d67-jztfn\" (UID: \"2fdd3964-fd57-4a4c-8706-7db9f2de6e47\") " pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"
Sep 05 02:51:59.712675 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.712501    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-proxy-tls\") pod \"error-404-isvc-a4a3e-predictor-6667499d67-jztfn\" (UID: \"2fdd3964-fd57-4a4c-8706-7db9f2de6e47\") " pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"
Sep 05 02:51:59.712675 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.712565    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x989\" (UniqueName: \"kubernetes.io/projected/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-kube-api-access-2x989\") pod \"error-404-isvc-a4a3e-predictor-6667499d67-jztfn\" (UID: \"2fdd3964-fd57-4a4c-8706-7db9f2de6e47\") " pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"
Sep 05 02:51:59.712791 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:51:59.712680    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-serving-cert: secret "error-404-isvc-a4a3e-predictor-serving-cert" not found
Sep 05 02:51:59.712791 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:51:59.712752    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-proxy-tls podName:2fdd3964-fd57-4a4c-8706-7db9f2de6e47 nodeName:}" failed. No retries permitted until 2026-09-05 02:52:00.212729252 +0000 UTC m=+2051.692123405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-proxy-tls") pod "error-404-isvc-a4a3e-predictor-6667499d67-jztfn" (UID: "2fdd3964-fd57-4a4c-8706-7db9f2de6e47") : secret "error-404-isvc-a4a3e-predictor-serving-cert" not found
Sep 05 02:51:59.713205 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.713185    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a4a3e-predictor-6667499d67-jztfn\" (UID: \"2fdd3964-fd57-4a4c-8706-7db9f2de6e47\") " pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"
Sep 05 02:51:59.722483 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:51:59.722458    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x989\" (UniqueName: \"kubernetes.io/projected/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-kube-api-access-2x989\") pod \"error-404-isvc-a4a3e-predictor-6667499d67-jztfn\" (UID: \"2fdd3964-fd57-4a4c-8706-7db9f2de6e47\") " pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"
Sep 05 02:52:00.217088 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:00.217049    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-proxy-tls\") pod \"error-404-isvc-a4a3e-predictor-6667499d67-jztfn\" (UID: \"2fdd3964-fd57-4a4c-8706-7db9f2de6e47\") " pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"
Sep 05 02:52:00.217484 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:52:00.217217    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-serving-cert: secret "error-404-isvc-a4a3e-predictor-serving-cert" not found
Sep 05 02:52:00.217484 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:52:00.217298    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-proxy-tls podName:2fdd3964-fd57-4a4c-8706-7db9f2de6e47 nodeName:}" failed. No retries permitted until 2026-09-05 02:52:01.217274981 +0000 UTC m=+2052.696669127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-proxy-tls") pod "error-404-isvc-a4a3e-predictor-6667499d67-jztfn" (UID: "2fdd3964-fd57-4a4c-8706-7db9f2de6e47") : secret "error-404-isvc-a4a3e-predictor-serving-cert" not found
Sep 05 02:52:00.346009 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:00.345956    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:52:01.223725 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:01.223686    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-proxy-tls\") pod \"error-404-isvc-a4a3e-predictor-6667499d67-jztfn\" (UID: \"2fdd3964-fd57-4a4c-8706-7db9f2de6e47\") " pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"
Sep 05 02:52:01.226023 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:01.226003    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-proxy-tls\") pod \"error-404-isvc-a4a3e-predictor-6667499d67-jztfn\" (UID: \"2fdd3964-fd57-4a4c-8706-7db9f2de6e47\") " pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"
Sep 05 02:52:01.297844 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:01.297800    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"
Sep 05 02:52:01.570264 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:01.570188    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"]
Sep 05 02:52:01.572882 ip-10-0-130-182 kubenswrapper[2566]: W0905 02:52:01.572857    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fdd3964_fd57_4a4c_8706_7db9f2de6e47.slice/crio-058b12be79bc437975e7f27688511ddd838bfe2f4f23efa1e798582bfecbd520 WatchSource:0}: Error finding container 058b12be79bc437975e7f27688511ddd838bfe2f4f23efa1e798582bfecbd520: Status 404 returned error can't find the container with id 058b12be79bc437975e7f27688511ddd838bfe2f4f23efa1e798582bfecbd520
Sep 05 02:52:02.566694 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:02.566661    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn" event={"ID":"2fdd3964-fd57-4a4c-8706-7db9f2de6e47","Type":"ContainerStarted","Data":"22075bab55ad168b8a8385f5f8d97af32226cbc9af32a3727a090092028aa279"}
Sep 05 02:52:02.567101 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:02.566721    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn" event={"ID":"2fdd3964-fd57-4a4c-8706-7db9f2de6e47","Type":"ContainerStarted","Data":"f677885c3c201694dd7964ed183bd1aa9ce635f2ce1fc61f88069a5eff716459"}
Sep 05 02:52:02.567101 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:02.566746    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"
Sep 05 02:52:02.567101 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:02.566765    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn" event={"ID":"2fdd3964-fd57-4a4c-8706-7db9f2de6e47","Type":"ContainerStarted","Data":"058b12be79bc437975e7f27688511ddd838bfe2f4f23efa1e798582bfecbd520"}
Sep 05 02:52:02.589020 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:02.588969    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn" podStartSLOduration=3.5889533780000002 podStartE2EDuration="3.588953378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:52:02.587521276 +0000 UTC m=+2054.066915433" watchObservedRunningTime="2026-09-05 02:52:02.588953378 +0000 UTC m=+2054.068347532"
Sep 05 02:52:02.825700 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:02.825646    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"
Sep 05 02:52:02.939848 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:02.939801    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd23f079-4a3b-4bbd-a05a-d5e22030d087-proxy-tls\") pod \"dd23f079-4a3b-4bbd-a05a-d5e22030d087\" (UID: \"dd23f079-4a3b-4bbd-a05a-d5e22030d087\") "
Sep 05 02:52:02.940001 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:02.939881    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd23f079-4a3b-4bbd-a05a-d5e22030d087-kube-rbac-proxy-sar-config\") pod \"dd23f079-4a3b-4bbd-a05a-d5e22030d087\" (UID: \"dd23f079-4a3b-4bbd-a05a-d5e22030d087\") "
Sep 05 02:52:02.940001 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:02.939921    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5vtz\" (UniqueName: \"kubernetes.io/projected/dd23f079-4a3b-4bbd-a05a-d5e22030d087-kube-api-access-n5vtz\") pod \"dd23f079-4a3b-4bbd-a05a-d5e22030d087\" (UID: \"dd23f079-4a3b-4bbd-a05a-d5e22030d087\") "
Sep 05 02:52:02.940224 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:02.940198    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd23f079-4a3b-4bbd-a05a-d5e22030d087-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dd23f079-4a3b-4bbd-a05a-d5e22030d087" (UID: "dd23f079-4a3b-4bbd-a05a-d5e22030d087"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:52:02.941919 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:02.941894    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd23f079-4a3b-4bbd-a05a-d5e22030d087-kube-api-access-n5vtz" (OuterVolumeSpecName: "kube-api-access-n5vtz") pod "dd23f079-4a3b-4bbd-a05a-d5e22030d087" (UID: "dd23f079-4a3b-4bbd-a05a-d5e22030d087"). InnerVolumeSpecName "kube-api-access-n5vtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:52:02.942011 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:02.941958    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd23f079-4a3b-4bbd-a05a-d5e22030d087-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dd23f079-4a3b-4bbd-a05a-d5e22030d087" (UID: "dd23f079-4a3b-4bbd-a05a-d5e22030d087"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:52:03.040348 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:03.040317    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5vtz\" (UniqueName: \"kubernetes.io/projected/dd23f079-4a3b-4bbd-a05a-d5e22030d087-kube-api-access-n5vtz\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:52:03.040348 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:03.040341    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd23f079-4a3b-4bbd-a05a-d5e22030d087-proxy-tls\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:52:03.040348 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:03.040351    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd23f079-4a3b-4bbd-a05a-d5e22030d087-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 02:52:03.572826 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:03.572790    2566 generic.go:356] "Generic (PLEG): container finished" podID="dd23f079-4a3b-4bbd-a05a-d5e22030d087" containerID="1cc1eaea7cb53c4e68a674d24a7b5e22363b23176cb98c25960f01018de1ff97" exitCode=0
Sep 05 02:52:03.573411 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:03.572854    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll" event={"ID":"dd23f079-4a3b-4bbd-a05a-d5e22030d087","Type":"ContainerDied","Data":"1cc1eaea7cb53c4e68a674d24a7b5e22363b23176cb98c25960f01018de1ff97"}
Sep 05 02:52:03.573411 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:03.572878    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"
Sep 05 02:52:03.573411 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:03.572924    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll" event={"ID":"dd23f079-4a3b-4bbd-a05a-d5e22030d087","Type":"ContainerDied","Data":"da2146835457c01ff21ca40e718d1bfda3a1c522ab199c7b8f0bfae8d35a5b01"}
Sep 05 02:52:03.573411 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:03.572958    2566 scope.go:117] "RemoveContainer" containerID="ddfe7c885ce71c0bd84919331ff3326d3840530e1d8acc62ef1eb9351aebea82"
Sep 05 02:52:03.573594 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:03.573438    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"
Sep 05 02:52:03.575191 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:03.575159    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn" podUID="2fdd3964-fd57-4a4c-8706-7db9f2de6e47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 05 02:52:03.581124 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:03.581106    2566 scope.go:117] "RemoveContainer" containerID="1cc1eaea7cb53c4e68a674d24a7b5e22363b23176cb98c25960f01018de1ff97"
Sep 05 02:52:03.588655 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:03.588608    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"]
Sep 05 02:52:03.588741 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:03.588725    2566 scope.go:117] "RemoveContainer" containerID="ddfe7c885ce71c0bd84919331ff3326d3840530e1d8acc62ef1eb9351aebea82"
Sep 05 02:52:03.589127 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:52:03.589100    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddfe7c885ce71c0bd84919331ff3326d3840530e1d8acc62ef1eb9351aebea82\": container with ID starting with ddfe7c885ce71c0bd84919331ff3326d3840530e1d8acc62ef1eb9351aebea82 not found: ID does not exist" containerID="ddfe7c885ce71c0bd84919331ff3326d3840530e1d8acc62ef1eb9351aebea82"
Sep 05 02:52:03.589235 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:03.589135    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddfe7c885ce71c0bd84919331ff3326d3840530e1d8acc62ef1eb9351aebea82"} err="failed to get container status \"ddfe7c885ce71c0bd84919331ff3326d3840530e1d8acc62ef1eb9351aebea82\": rpc error: code = NotFound desc = could not find container \"ddfe7c885ce71c0bd84919331ff3326d3840530e1d8acc62ef1eb9351aebea82\": container with ID starting with ddfe7c885ce71c0bd84919331ff3326d3840530e1d8acc62ef1eb9351aebea82 not found: ID does not exist"
Sep 05 02:52:03.589235 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:03.589158    2566 scope.go:117] "RemoveContainer" containerID="1cc1eaea7cb53c4e68a674d24a7b5e22363b23176cb98c25960f01018de1ff97"
Sep 05 02:52:03.589483 ip-10-0-130-182 kubenswrapper[2566]: E0905 02:52:03.589452    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cc1eaea7cb53c4e68a674d24a7b5e22363b23176cb98c25960f01018de1ff97\": container with ID starting with 1cc1eaea7cb53c4e68a674d24a7b5e22363b23176cb98c25960f01018de1ff97 not found: ID does not exist" containerID="1cc1eaea7cb53c4e68a674d24a7b5e22363b23176cb98c25960f01018de1ff97"
Sep 05 02:52:03.589530 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:03.589479    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cc1eaea7cb53c4e68a674d24a7b5e22363b23176cb98c25960f01018de1ff97"} err="failed to get container status \"1cc1eaea7cb53c4e68a674d24a7b5e22363b23176cb98c25960f01018de1ff97\": rpc error: code = NotFound desc = could not find container \"1cc1eaea7cb53c4e68a674d24a7b5e22363b23176cb98c25960f01018de1ff97\": container with ID starting with 1cc1eaea7cb53c4e68a674d24a7b5e22363b23176cb98c25960f01018de1ff97 not found: ID does not exist"
Sep 05 02:52:03.590157 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:03.590139    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ca9bf-predictor-5bbff7d989-mn5ll"]
Sep 05 02:52:04.577996 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:04.577954    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn" podUID="2fdd3964-fd57-4a4c-8706-7db9f2de6e47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 05 02:52:05.047827 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:05.047793    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd23f079-4a3b-4bbd-a05a-d5e22030d087" path="/var/lib/kubelet/pods/dd23f079-4a3b-4bbd-a05a-d5e22030d087/volumes"
Sep 05 02:52:07.531173 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:07.531131    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4" podUID="ae4118a5-a694-49d0-92d9-d4f154d69c57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 05 02:52:09.582292 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:09.582263    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"
Sep 05 02:52:09.582781 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:09.582741    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn" podUID="2fdd3964-fd57-4a4c-8706-7db9f2de6e47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 05 02:52:17.530999 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:17.530914    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4" podUID="ae4118a5-a694-49d0-92d9-d4f154d69c57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 05 02:52:19.583315 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:19.583275    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn" podUID="2fdd3964-fd57-4a4c-8706-7db9f2de6e47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 05 02:52:27.530961 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:27.530918    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4" podUID="ae4118a5-a694-49d0-92d9-d4f154d69c57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 05 02:52:29.582781 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:29.582743    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn" podUID="2fdd3964-fd57-4a4c-8706-7db9f2de6e47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 05 02:52:37.531800 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:37.531769    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"
Sep 05 02:52:39.582982 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:39.582939    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn" podUID="2fdd3964-fd57-4a4c-8706-7db9f2de6e47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 05 02:52:48.957376 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:48.957344    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:52:49.583891 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:49.583862    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"
Sep 05 02:52:50.815997 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:50.815964    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:52:59.399806 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:52:59.399768    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:53:00.111844 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:53:00.111803    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:54:20.292690 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:54:20.292656    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:54:29.306666 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:54:29.306632    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 02:59:22.258694 ip-10-0-130-182 kubenswrapper[2566]: I0905 02:59:22.258662    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 03:01:14.016822 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:14.016739    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 03:01:14.138475 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:14.138436    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 03:01:14.144614 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:14.144587    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"]
Sep 05 03:01:14.144879 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:14.144856    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn" podUID="2fdd3964-fd57-4a4c-8706-7db9f2de6e47" containerName="kserve-container" containerID="cri-o://f677885c3c201694dd7964ed183bd1aa9ce635f2ce1fc61f88069a5eff716459" gracePeriod=30
Sep 05 03:01:14.144982 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:14.144851    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn" podUID="2fdd3964-fd57-4a4c-8706-7db9f2de6e47" containerName="kube-rbac-proxy" containerID="cri-o://22075bab55ad168b8a8385f5f8d97af32226cbc9af32a3727a090092028aa279" gracePeriod=30
Sep 05 03:01:14.213215 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:14.213170    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 03:01:14.566984 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:14.566951    2566 generic.go:356] "Generic (PLEG): container finished" podID="2fdd3964-fd57-4a4c-8706-7db9f2de6e47" containerID="22075bab55ad168b8a8385f5f8d97af32226cbc9af32a3727a090092028aa279" exitCode=2
Sep 05 03:01:14.567149 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:14.567026    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn" event={"ID":"2fdd3964-fd57-4a4c-8706-7db9f2de6e47","Type":"ContainerDied","Data":"22075bab55ad168b8a8385f5f8d97af32226cbc9af32a3727a090092028aa279"}
Sep 05 03:01:14.578465 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:14.578432    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn" podUID="2fdd3964-fd57-4a4c-8706-7db9f2de6e47" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 05 03:01:17.479372 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:17.479346    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"
Sep 05 03:01:17.575937 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:17.575825    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2x989\" (UniqueName: \"kubernetes.io/projected/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-kube-api-access-2x989\") pod \"2fdd3964-fd57-4a4c-8706-7db9f2de6e47\" (UID: \"2fdd3964-fd57-4a4c-8706-7db9f2de6e47\") "
Sep 05 03:01:17.575937 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:17.575882    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-kube-rbac-proxy-sar-config\") pod \"2fdd3964-fd57-4a4c-8706-7db9f2de6e47\" (UID: \"2fdd3964-fd57-4a4c-8706-7db9f2de6e47\") "
Sep 05 03:01:17.575937 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:17.575922    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-proxy-tls\") pod \"2fdd3964-fd57-4a4c-8706-7db9f2de6e47\" (UID: \"2fdd3964-fd57-4a4c-8706-7db9f2de6e47\") "
Sep 05 03:01:17.576310 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:17.576271    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2fdd3964-fd57-4a4c-8706-7db9f2de6e47" (UID: "2fdd3964-fd57-4a4c-8706-7db9f2de6e47"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:01:17.578092 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:17.578066    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-kube-api-access-2x989" (OuterVolumeSpecName: "kube-api-access-2x989") pod "2fdd3964-fd57-4a4c-8706-7db9f2de6e47" (UID: "2fdd3964-fd57-4a4c-8706-7db9f2de6e47"). InnerVolumeSpecName "kube-api-access-2x989". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:01:17.578370 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:17.578345    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2fdd3964-fd57-4a4c-8706-7db9f2de6e47" (UID: "2fdd3964-fd57-4a4c-8706-7db9f2de6e47"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:01:17.580316 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:17.580291    2566 generic.go:356] "Generic (PLEG): container finished" podID="2fdd3964-fd57-4a4c-8706-7db9f2de6e47" containerID="f677885c3c201694dd7964ed183bd1aa9ce635f2ce1fc61f88069a5eff716459" exitCode=0
Sep 05 03:01:17.580398 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:17.580327    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn" event={"ID":"2fdd3964-fd57-4a4c-8706-7db9f2de6e47","Type":"ContainerDied","Data":"f677885c3c201694dd7964ed183bd1aa9ce635f2ce1fc61f88069a5eff716459"}
Sep 05 03:01:17.580398 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:17.580358    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"
Sep 05 03:01:17.580476 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:17.580407    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn" event={"ID":"2fdd3964-fd57-4a4c-8706-7db9f2de6e47","Type":"ContainerDied","Data":"058b12be79bc437975e7f27688511ddd838bfe2f4f23efa1e798582bfecbd520"}
Sep 05 03:01:17.580476 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:17.580435    2566 scope.go:117] "RemoveContainer" containerID="22075bab55ad168b8a8385f5f8d97af32226cbc9af32a3727a090092028aa279"
Sep 05 03:01:17.591406 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:17.591384    2566 scope.go:117] "RemoveContainer" containerID="f677885c3c201694dd7964ed183bd1aa9ce635f2ce1fc61f88069a5eff716459"
Sep 05 03:01:17.601258 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:17.601236    2566 scope.go:117] "RemoveContainer" containerID="22075bab55ad168b8a8385f5f8d97af32226cbc9af32a3727a090092028aa279"
Sep 05 03:01:17.601564 ip-10-0-130-182 kubenswrapper[2566]: E0905 03:01:17.601545    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22075bab55ad168b8a8385f5f8d97af32226cbc9af32a3727a090092028aa279\": container with ID starting with 22075bab55ad168b8a8385f5f8d97af32226cbc9af32a3727a090092028aa279 not found: ID does not exist" containerID="22075bab55ad168b8a8385f5f8d97af32226cbc9af32a3727a090092028aa279"
Sep 05 03:01:17.601613 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:17.601574    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22075bab55ad168b8a8385f5f8d97af32226cbc9af32a3727a090092028aa279"} err="failed to get container status \"22075bab55ad168b8a8385f5f8d97af32226cbc9af32a3727a090092028aa279\": rpc error: code = NotFound desc = could not find container \"22075bab55ad168b8a8385f5f8d97af32226cbc9af32a3727a090092028aa279\": container with ID starting with 22075bab55ad168b8a8385f5f8d97af32226cbc9af32a3727a090092028aa279 not found: ID does not exist"
Sep 05 03:01:17.601613 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:17.601593    2566 scope.go:117] "RemoveContainer" containerID="f677885c3c201694dd7964ed183bd1aa9ce635f2ce1fc61f88069a5eff716459"
Sep 05 03:01:17.601880 ip-10-0-130-182 kubenswrapper[2566]: E0905 03:01:17.601860    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f677885c3c201694dd7964ed183bd1aa9ce635f2ce1fc61f88069a5eff716459\": container with ID starting with f677885c3c201694dd7964ed183bd1aa9ce635f2ce1fc61f88069a5eff716459 not found: ID does not exist" containerID="f677885c3c201694dd7964ed183bd1aa9ce635f2ce1fc61f88069a5eff716459"
Sep 05 03:01:17.601943 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:17.601885    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f677885c3c201694dd7964ed183bd1aa9ce635f2ce1fc61f88069a5eff716459"} err="failed to get container status \"f677885c3c201694dd7964ed183bd1aa9ce635f2ce1fc61f88069a5eff716459\": rpc error: code = NotFound desc = could not find container \"f677885c3c201694dd7964ed183bd1aa9ce635f2ce1fc61f88069a5eff716459\": container with ID starting with f677885c3c201694dd7964ed183bd1aa9ce635f2ce1fc61f88069a5eff716459 not found: ID does not exist"
Sep 05 03:01:17.604857 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:17.604819    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"]
Sep 05 03:01:17.608113 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:17.608093    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a4a3e-predictor-6667499d67-jztfn"]
Sep 05 03:01:17.676412 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:17.676380    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-proxy-tls\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 03:01:17.676412 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:17.676411    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x989\" (UniqueName: \"kubernetes.io/projected/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-kube-api-access-2x989\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 03:01:17.676602 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:17.676427    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fdd3964-fd57-4a4c-8706-7db9f2de6e47-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 03:01:19.047746 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:01:19.047703    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fdd3964-fd57-4a4c-8706-7db9f2de6e47" path="/var/lib/kubelet/pods/2fdd3964-fd57-4a4c-8706-7db9f2de6e47/volumes"
Sep 05 03:04:28.581334 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:04:28.581242    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 03:04:40.761387 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:04:40.761353    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 03:09:07.897876 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:07.897825    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 03:09:08.002919 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:08.002877    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 03:09:08.036953 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:08.036917    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"]
Sep 05 03:09:08.037162 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:08.037141    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4" podUID="ae4118a5-a694-49d0-92d9-d4f154d69c57" containerName="kserve-container" containerID="cri-o://7d38953c8e00e0bd7bd53c165870bd7c60bedaec12f04e896fbf261be69c7d6a" gracePeriod=30
Sep 05 03:09:08.037290 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:08.037170    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4" podUID="ae4118a5-a694-49d0-92d9-d4f154d69c57" containerName="kube-rbac-proxy" containerID="cri-o://a7edf3822d16c7db7ff5348ceba1e242bb3fd5912caa0c6d8fea37440720f86c" gracePeriod=30
Sep 05 03:09:08.101118 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:08.101077    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 03:09:08.237281 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:08.237249    2566 generic.go:356] "Generic (PLEG): container finished" podID="ae4118a5-a694-49d0-92d9-d4f154d69c57" containerID="a7edf3822d16c7db7ff5348ceba1e242bb3fd5912caa0c6d8fea37440720f86c" exitCode=2
Sep 05 03:09:08.237437 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:08.237324    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4" event={"ID":"ae4118a5-a694-49d0-92d9-d4f154d69c57","Type":"ContainerDied","Data":"a7edf3822d16c7db7ff5348ceba1e242bb3fd5912caa0c6d8fea37440720f86c"}
Sep 05 03:09:08.832769 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:08.832731    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 03:09:08.926811 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:08.926777    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 03:09:11.070097 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:11.070071    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"
Sep 05 03:09:11.193371 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:11.193280    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae4118a5-a694-49d0-92d9-d4f154d69c57-proxy-tls\") pod \"ae4118a5-a694-49d0-92d9-d4f154d69c57\" (UID: \"ae4118a5-a694-49d0-92d9-d4f154d69c57\") "
Sep 05 03:09:11.193371 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:11.193364    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52ndl\" (UniqueName: \"kubernetes.io/projected/ae4118a5-a694-49d0-92d9-d4f154d69c57-kube-api-access-52ndl\") pod \"ae4118a5-a694-49d0-92d9-d4f154d69c57\" (UID: \"ae4118a5-a694-49d0-92d9-d4f154d69c57\") "
Sep 05 03:09:11.193608 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:11.193388    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae4118a5-a694-49d0-92d9-d4f154d69c57-kube-rbac-proxy-sar-config\") pod \"ae4118a5-a694-49d0-92d9-d4f154d69c57\" (UID: \"ae4118a5-a694-49d0-92d9-d4f154d69c57\") "
Sep 05 03:09:11.193879 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:11.193828    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae4118a5-a694-49d0-92d9-d4f154d69c57-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae4118a5-a694-49d0-92d9-d4f154d69c57" (UID: "ae4118a5-a694-49d0-92d9-d4f154d69c57"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:09:11.195459 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:11.195439    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4118a5-a694-49d0-92d9-d4f154d69c57-kube-api-access-52ndl" (OuterVolumeSpecName: "kube-api-access-52ndl") pod "ae4118a5-a694-49d0-92d9-d4f154d69c57" (UID: "ae4118a5-a694-49d0-92d9-d4f154d69c57"). InnerVolumeSpecName "kube-api-access-52ndl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:09:11.195569 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:11.195529    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae4118a5-a694-49d0-92d9-d4f154d69c57-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae4118a5-a694-49d0-92d9-d4f154d69c57" (UID: "ae4118a5-a694-49d0-92d9-d4f154d69c57"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:09:11.249205 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:11.249170    2566 generic.go:356] "Generic (PLEG): container finished" podID="ae4118a5-a694-49d0-92d9-d4f154d69c57" containerID="7d38953c8e00e0bd7bd53c165870bd7c60bedaec12f04e896fbf261be69c7d6a" exitCode=0
Sep 05 03:09:11.249386 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:11.249218    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"
Sep 05 03:09:11.249386 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:11.249262    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4" event={"ID":"ae4118a5-a694-49d0-92d9-d4f154d69c57","Type":"ContainerDied","Data":"7d38953c8e00e0bd7bd53c165870bd7c60bedaec12f04e896fbf261be69c7d6a"}
Sep 05 03:09:11.249386 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:11.249337    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4" event={"ID":"ae4118a5-a694-49d0-92d9-d4f154d69c57","Type":"ContainerDied","Data":"287c95be279ef842618f1e9bb06357be1afc8443614d5c40106c1f553a8a4a07"}
Sep 05 03:09:11.249386 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:11.249373    2566 scope.go:117] "RemoveContainer" containerID="a7edf3822d16c7db7ff5348ceba1e242bb3fd5912caa0c6d8fea37440720f86c"
Sep 05 03:09:11.257392 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:11.257376    2566 scope.go:117] "RemoveContainer" containerID="7d38953c8e00e0bd7bd53c165870bd7c60bedaec12f04e896fbf261be69c7d6a"
Sep 05 03:09:11.264167 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:11.264151    2566 scope.go:117] "RemoveContainer" containerID="a7edf3822d16c7db7ff5348ceba1e242bb3fd5912caa0c6d8fea37440720f86c"
Sep 05 03:09:11.264417 ip-10-0-130-182 kubenswrapper[2566]: E0905 03:09:11.264402    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7edf3822d16c7db7ff5348ceba1e242bb3fd5912caa0c6d8fea37440720f86c\": container with ID starting with a7edf3822d16c7db7ff5348ceba1e242bb3fd5912caa0c6d8fea37440720f86c not found: ID does not exist" containerID="a7edf3822d16c7db7ff5348ceba1e242bb3fd5912caa0c6d8fea37440720f86c"
Sep 05 03:09:11.264466 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:11.264424    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7edf3822d16c7db7ff5348ceba1e242bb3fd5912caa0c6d8fea37440720f86c"} err="failed to get container status \"a7edf3822d16c7db7ff5348ceba1e242bb3fd5912caa0c6d8fea37440720f86c\": rpc error: code = NotFound desc = could not find container \"a7edf3822d16c7db7ff5348ceba1e242bb3fd5912caa0c6d8fea37440720f86c\": container with ID starting with a7edf3822d16c7db7ff5348ceba1e242bb3fd5912caa0c6d8fea37440720f86c not found: ID does not exist"
Sep 05 03:09:11.264466 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:11.264442    2566 scope.go:117] "RemoveContainer" containerID="7d38953c8e00e0bd7bd53c165870bd7c60bedaec12f04e896fbf261be69c7d6a"
Sep 05 03:09:11.264682 ip-10-0-130-182 kubenswrapper[2566]: E0905 03:09:11.264665    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d38953c8e00e0bd7bd53c165870bd7c60bedaec12f04e896fbf261be69c7d6a\": container with ID starting with 7d38953c8e00e0bd7bd53c165870bd7c60bedaec12f04e896fbf261be69c7d6a not found: ID does not exist" containerID="7d38953c8e00e0bd7bd53c165870bd7c60bedaec12f04e896fbf261be69c7d6a"
Sep 05 03:09:11.264738 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:11.264692    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d38953c8e00e0bd7bd53c165870bd7c60bedaec12f04e896fbf261be69c7d6a"} err="failed to get container status \"7d38953c8e00e0bd7bd53c165870bd7c60bedaec12f04e896fbf261be69c7d6a\": rpc error: code = NotFound desc = could not find container \"7d38953c8e00e0bd7bd53c165870bd7c60bedaec12f04e896fbf261be69c7d6a\": container with ID starting with 7d38953c8e00e0bd7bd53c165870bd7c60bedaec12f04e896fbf261be69c7d6a not found: ID does not exist"
Sep 05 03:09:11.268994 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:11.268968    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"]
Sep 05 03:09:11.274945 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:11.274926    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3af6d-predictor-7dd87fb4f9-zgxk4"]
Sep 05 03:09:11.293809 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:11.293785    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52ndl\" (UniqueName: \"kubernetes.io/projected/ae4118a5-a694-49d0-92d9-d4f154d69c57-kube-api-access-52ndl\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 03:09:11.293809 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:11.293807    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae4118a5-a694-49d0-92d9-d4f154d69c57-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 03:09:11.293938 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:11.293820    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae4118a5-a694-49d0-92d9-d4f154d69c57-proxy-tls\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Sep 05 03:09:13.047035 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:13.046996    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae4118a5-a694-49d0-92d9-d4f154d69c57" path="/var/lib/kubelet/pods/ae4118a5-a694-49d0-92d9-d4f154d69c57/volumes"
Sep 05 03:09:37.426518 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.426472    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 03:09:37.568140 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.568102    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zkkfn/must-gather-pt2vf"]
Sep 05 03:09:37.568368 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.568356    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd23f079-4a3b-4bbd-a05a-d5e22030d087" containerName="kserve-container"
Sep 05 03:09:37.568416 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.568371    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd23f079-4a3b-4bbd-a05a-d5e22030d087" containerName="kserve-container"
Sep 05 03:09:37.568416 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.568381    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd23f079-4a3b-4bbd-a05a-d5e22030d087" containerName="kube-rbac-proxy"
Sep 05 03:09:37.568416 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.568387    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd23f079-4a3b-4bbd-a05a-d5e22030d087" containerName="kube-rbac-proxy"
Sep 05 03:09:37.568416 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.568395    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fdd3964-fd57-4a4c-8706-7db9f2de6e47" containerName="kube-rbac-proxy"
Sep 05 03:09:37.568416 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.568401    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fdd3964-fd57-4a4c-8706-7db9f2de6e47" containerName="kube-rbac-proxy"
Sep 05 03:09:37.568416 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.568410    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae4118a5-a694-49d0-92d9-d4f154d69c57" containerName="kube-rbac-proxy"
Sep 05 03:09:37.568416 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.568415    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae4118a5-a694-49d0-92d9-d4f154d69c57" containerName="kube-rbac-proxy"
Sep 05 03:09:37.568635 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.568423    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fdd3964-fd57-4a4c-8706-7db9f2de6e47" containerName="kserve-container"
Sep 05 03:09:37.568635 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.568429    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fdd3964-fd57-4a4c-8706-7db9f2de6e47" containerName="kserve-container"
Sep 05 03:09:37.568635 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.568439    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae4118a5-a694-49d0-92d9-d4f154d69c57" containerName="kserve-container"
Sep 05 03:09:37.568635 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.568444    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae4118a5-a694-49d0-92d9-d4f154d69c57" containerName="kserve-container"
Sep 05 03:09:37.568635 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.568484    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd23f079-4a3b-4bbd-a05a-d5e22030d087" containerName="kube-rbac-proxy"
Sep 05 03:09:37.568635 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.568490    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae4118a5-a694-49d0-92d9-d4f154d69c57" containerName="kube-rbac-proxy"
Sep 05 03:09:37.568635 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.568496    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd23f079-4a3b-4bbd-a05a-d5e22030d087" containerName="kserve-container"
Sep 05 03:09:37.568635 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.568501    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fdd3964-fd57-4a4c-8706-7db9f2de6e47" containerName="kube-rbac-proxy"
Sep 05 03:09:37.568635 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.568507    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fdd3964-fd57-4a4c-8706-7db9f2de6e47" containerName="kserve-container"
Sep 05 03:09:37.568635 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.568511    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae4118a5-a694-49d0-92d9-d4f154d69c57" containerName="kserve-container"
Sep 05 03:09:37.571213 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.571196    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zkkfn/must-gather-pt2vf"
Sep 05 03:09:37.573582 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.573554    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zkkfn\"/\"openshift-service-ca.crt\""
Sep 05 03:09:37.573582 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.573578    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zkkfn\"/\"kube-root-ca.crt\""
Sep 05 03:09:37.574434 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.574412    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zkkfn\"/\"default-dockercfg-5rzbt\""
Sep 05 03:09:37.579541 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.579512    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zkkfn/must-gather-pt2vf"]
Sep 05 03:09:37.693258 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.693166    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c7f259e9-0a41-4588-9c50-655816d66340-must-gather-output\") pod \"must-gather-pt2vf\" (UID: \"c7f259e9-0a41-4588-9c50-655816d66340\") " pod="openshift-must-gather-zkkfn/must-gather-pt2vf"
Sep 05 03:09:37.693258 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.693205    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q4ss\" (UniqueName: \"kubernetes.io/projected/c7f259e9-0a41-4588-9c50-655816d66340-kube-api-access-8q4ss\") pod \"must-gather-pt2vf\" (UID: \"c7f259e9-0a41-4588-9c50-655816d66340\") " pod="openshift-must-gather-zkkfn/must-gather-pt2vf"
Sep 05 03:09:37.794203 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.794149    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c7f259e9-0a41-4588-9c50-655816d66340-must-gather-output\") pod \"must-gather-pt2vf\" (UID: \"c7f259e9-0a41-4588-9c50-655816d66340\") " pod="openshift-must-gather-zkkfn/must-gather-pt2vf"
Sep 05 03:09:37.794203 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.794204    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q4ss\" (UniqueName: \"kubernetes.io/projected/c7f259e9-0a41-4588-9c50-655816d66340-kube-api-access-8q4ss\") pod \"must-gather-pt2vf\" (UID: \"c7f259e9-0a41-4588-9c50-655816d66340\") " pod="openshift-must-gather-zkkfn/must-gather-pt2vf"
Sep 05 03:09:37.794494 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.794476    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c7f259e9-0a41-4588-9c50-655816d66340-must-gather-output\") pod \"must-gather-pt2vf\" (UID: \"c7f259e9-0a41-4588-9c50-655816d66340\") " pod="openshift-must-gather-zkkfn/must-gather-pt2vf"
Sep 05 03:09:37.816231 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.816199    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q4ss\" (UniqueName: \"kubernetes.io/projected/c7f259e9-0a41-4588-9c50-655816d66340-kube-api-access-8q4ss\") pod \"must-gather-pt2vf\" (UID: \"c7f259e9-0a41-4588-9c50-655816d66340\") " pod="openshift-must-gather-zkkfn/must-gather-pt2vf"
Sep 05 03:09:37.880387 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:37.880343    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zkkfn/must-gather-pt2vf"
Sep 05 03:09:38.153484 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:38.150929    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zkkfn/must-gather-pt2vf"]
Sep 05 03:09:38.157251 ip-10-0-130-182 kubenswrapper[2566]: W0905 03:09:38.157222    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7f259e9_0a41_4588_9c50_655816d66340.slice/crio-2664cd0e79029a22d1f08c41df1d331aa4f67ffe984678756dac8344d339659e WatchSource:0}: Error finding container 2664cd0e79029a22d1f08c41df1d331aa4f67ffe984678756dac8344d339659e: Status 404 returned error can't find the container with id 2664cd0e79029a22d1f08c41df1d331aa4f67ffe984678756dac8344d339659e
Sep 05 03:09:38.159014 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:38.158996    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 03:09:38.344781 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:38.344740    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zkkfn/must-gather-pt2vf" event={"ID":"c7f259e9-0a41-4588-9c50-655816d66340","Type":"ContainerStarted","Data":"2664cd0e79029a22d1f08c41df1d331aa4f67ffe984678756dac8344d339659e"}
Sep 05 03:09:38.610734 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:38.610647    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 03:09:39.351723 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:39.351685    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zkkfn/must-gather-pt2vf" event={"ID":"c7f259e9-0a41-4588-9c50-655816d66340","Type":"ContainerStarted","Data":"ab3c53e7d8450cfa178655274949e4b1268f21110583a568b04632a70f4f6f97"}
Sep 05 03:09:39.351924 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:39.351733    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zkkfn/must-gather-pt2vf" event={"ID":"c7f259e9-0a41-4588-9c50-655816d66340","Type":"ContainerStarted","Data":"5b62994793206835e0c6774d5a4d8a7a0542d44e769257f66add47d98877cfe0"}
Sep 05 03:09:39.371738 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:39.371674    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zkkfn/must-gather-pt2vf" podStartSLOduration=1.7686827520000001 podStartE2EDuration="2.371649039s" totalImagesPullingTime="602.966287ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:09:39.369827011 +0000 UTC m=+3110.849221168" watchObservedRunningTime="2026-09-05 03:09:39.371649039 +0000 UTC m=+3110.851043198"
Sep 05 03:09:40.265660 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:40.265629    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kj85s_37ca8378-c8ef-4342-b489-631cba71f8c5/global-pull-secret-syncer/0.log"
Sep 05 03:09:40.329725 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:40.329670    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log"
Sep 05 03:09:40.450824 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:40.450796    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-182.ec2.internal_d36b70f3f303d4cb4f8a7bad8d41d64d/haproxy/0.log"
Sep 05 03:09:43.071033 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:43.070982    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 03:09:43.173476 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:43.173437    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wxxsg_974e8bfc-b802-4dfa-94c6-75a4331be5a5/konnectivity-agent/0.log" limit=1024
Sep 05 03:09:45.026803 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:45.026769    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kw8ds_48cd7ba6-45a0-4508-a0dd-0a04e77ab363/node-exporter/0.log"
Sep 05 03:09:45.052786 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:45.052760    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kw8ds_48cd7ba6-45a0-4508-a0dd-0a04e77ab363/kube-rbac-proxy/0.log"
Sep 05 03:09:45.080868 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:45.080823    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kw8ds_48cd7ba6-45a0-4508-a0dd-0a04e77ab363/init-textfile/0.log"
Sep 05 03:09:48.097804 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:48.097773    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-52z87_4e6c1fd5-e86b-497b-b5a3-06616e4fd09b/download-server/0.log"
Sep 05 03:09:48.249757 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:48.249721    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb"]
Sep 05 03:09:48.254941 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:48.254916    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb"
Sep 05 03:09:48.267620 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:48.267595    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb"]
Sep 05 03:09:48.279350 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:48.279316    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3526476d-de11-4feb-b7b1-97c83967ebb6-proc\") pod \"perf-node-gather-daemonset-5g7bb\" (UID: \"3526476d-de11-4feb-b7b1-97c83967ebb6\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb"
Sep 05 03:09:48.279518 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:48.279389    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6ltf\" (UniqueName: \"kubernetes.io/projected/3526476d-de11-4feb-b7b1-97c83967ebb6-kube-api-access-s6ltf\") pod \"perf-node-gather-daemonset-5g7bb\" (UID: \"3526476d-de11-4feb-b7b1-97c83967ebb6\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb"
Sep 05 03:09:48.279518 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:48.279434    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3526476d-de11-4feb-b7b1-97c83967ebb6-podres\") pod \"perf-node-gather-daemonset-5g7bb\" (UID: \"3526476d-de11-4feb-b7b1-97c83967ebb6\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb"
Sep 05 03:09:48.279518 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:48.279466    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3526476d-de11-4feb-b7b1-97c83967ebb6-lib-modules\") pod \"perf-node-gather-daemonset-5g7bb\" (UID: \"3526476d-de11-4feb-b7b1-97c83967ebb6\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb"
Sep 05 03:09:48.279518 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:48.279492    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3526476d-de11-4feb-b7b1-97c83967ebb6-sys\") pod \"perf-node-gather-daemonset-5g7bb\" (UID: \"3526476d-de11-4feb-b7b1-97c83967ebb6\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb"
Sep 05 03:09:48.380702 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:48.380667    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3526476d-de11-4feb-b7b1-97c83967ebb6-sys\") pod \"perf-node-gather-daemonset-5g7bb\" (UID: \"3526476d-de11-4feb-b7b1-97c83967ebb6\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb"
Sep 05 03:09:48.380921 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:48.380708    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3526476d-de11-4feb-b7b1-97c83967ebb6-proc\") pod \"perf-node-gather-daemonset-5g7bb\" (UID: \"3526476d-de11-4feb-b7b1-97c83967ebb6\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb"
Sep 05 03:09:48.380921 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:48.380771    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6ltf\" (UniqueName: \"kubernetes.io/projected/3526476d-de11-4feb-b7b1-97c83967ebb6-kube-api-access-s6ltf\") pod \"perf-node-gather-daemonset-5g7bb\" (UID: \"3526476d-de11-4feb-b7b1-97c83967ebb6\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb"
Sep 05 03:09:48.380921 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:48.380793    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3526476d-de11-4feb-b7b1-97c83967ebb6-sys\") pod \"perf-node-gather-daemonset-5g7bb\" (UID: \"3526476d-de11-4feb-b7b1-97c83967ebb6\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb"
Sep 05 03:09:48.380921 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:48.380819    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3526476d-de11-4feb-b7b1-97c83967ebb6-podres\") pod \"perf-node-gather-daemonset-5g7bb\" (UID: \"3526476d-de11-4feb-b7b1-97c83967ebb6\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb"
Sep 05 03:09:48.380921 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:48.380873    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3526476d-de11-4feb-b7b1-97c83967ebb6-lib-modules\") pod \"perf-node-gather-daemonset-5g7bb\" (UID: \"3526476d-de11-4feb-b7b1-97c83967ebb6\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb"
Sep 05 03:09:48.380921 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:48.380909    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3526476d-de11-4feb-b7b1-97c83967ebb6-proc\") pod \"perf-node-gather-daemonset-5g7bb\" (UID: \"3526476d-de11-4feb-b7b1-97c83967ebb6\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb"
Sep 05 03:09:48.381288 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:48.381016    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3526476d-de11-4feb-b7b1-97c83967ebb6-lib-modules\") pod \"perf-node-gather-daemonset-5g7bb\" (UID: \"3526476d-de11-4feb-b7b1-97c83967ebb6\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb"
Sep 05 03:09:48.381288 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:48.381033    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3526476d-de11-4feb-b7b1-97c83967ebb6-podres\") pod \"perf-node-gather-daemonset-5g7bb\" (UID: \"3526476d-de11-4feb-b7b1-97c83967ebb6\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb"
Sep 05 03:09:48.389674 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:48.389641    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6ltf\" (UniqueName: \"kubernetes.io/projected/3526476d-de11-4feb-b7b1-97c83967ebb6-kube-api-access-s6ltf\") pod \"perf-node-gather-daemonset-5g7bb\" (UID: \"3526476d-de11-4feb-b7b1-97c83967ebb6\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb"
Sep 05 03:09:48.567143 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:48.567109    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb"
Sep 05 03:09:48.854527 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:48.854493    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb"]
Sep 05 03:09:48.859720 ip-10-0-130-182 kubenswrapper[2566]: W0905 03:09:48.859687    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3526476d_de11_4feb_b7b1_97c83967ebb6.slice/crio-5cd7b22ffdada0600aae6b0920d4f2b5550a90e09e2c8252330595179fb9a4a0 WatchSource:0}: Error finding container 5cd7b22ffdada0600aae6b0920d4f2b5550a90e09e2c8252330595179fb9a4a0: Status 404 returned error can't find the container with id 5cd7b22ffdada0600aae6b0920d4f2b5550a90e09e2c8252330595179fb9a4a0
Sep 05 03:09:49.408337 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:49.408298    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb" event={"ID":"3526476d-de11-4feb-b7b1-97c83967ebb6","Type":"ContainerStarted","Data":"0ce333cd35285af4ff2a4be3459a7cbba24fa23c257108e25af3aadc87592a74"}
Sep 05 03:09:49.408808 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:49.408393    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb" event={"ID":"3526476d-de11-4feb-b7b1-97c83967ebb6","Type":"ContainerStarted","Data":"5cd7b22ffdada0600aae6b0920d4f2b5550a90e09e2c8252330595179fb9a4a0"}
Sep 05 03:09:49.408808 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:49.408463    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb"
Sep 05 03:09:49.427603 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:49.426721    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb" podStartSLOduration=1.426699192 podStartE2EDuration="1.426699192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 03:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:09:49.425386883 +0000 UTC m=+3120.904781041" watchObservedRunningTime="2026-09-05 03:09:49.426699192 +0000 UTC m=+3120.906093351"
Sep 05 03:09:49.548036 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:49.547943    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-66mtm_18051063-ebe9-4362-ab51-4a4da32e15ff/dns/0.log"
Sep 05 03:09:49.572154 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:49.572126    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-66mtm_18051063-ebe9-4362-ab51-4a4da32e15ff/kube-rbac-proxy/0.log"
Sep 05 03:09:49.715979 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:49.715940    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b89gp_24ff133a-c6d2-4cbc-8dd8-a1e662c05baf/dns-node-resolver/0.log"
Sep 05 03:09:50.261945 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:50.261909    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vxccn_c290fb8c-2fc3-4aff-863a-d8bf1ee41e01/node-ca/0.log"
Sep 05 03:09:51.472726 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:51.472695    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zgfdl_6b81f383-6a65-49a2-a8d8-c5efe33ffe3d/serve-healthcheck-canary/0.log"
Sep 05 03:09:51.889387 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:51.889357    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b2fq5_a41caf21-6b3f-4804-a45a-ed0ebec12315/kube-rbac-proxy/0.log"
Sep 05 03:09:51.919486 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:51.919456    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b2fq5_a41caf21-6b3f-4804-a45a-ed0ebec12315/exporter/0.log"
Sep 05 03:09:51.948640 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:51.948616    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b2fq5_a41caf21-6b3f-4804-a45a-ed0ebec12315/extractor/0.log"
Sep 05 03:09:54.378819 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:54.378784    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-577b5fb6f8-x6g4d_d5d218bf-9dd4-40f8-a91c-b61bbbb65782/manager/0.log"
Sep 05 03:09:54.419033 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:54.418997    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-mtxv7_0a1b84cf-6d08-4fc5-8356-c6a19338686f/manager/0.log"
Sep 05 03:09:55.422621 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:55.422585    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-5g7bb"
Sep 05 03:09:58.959447 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:58.959413    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-s54l4_f68966fc-2067-46e7-a102-aa8af80ad1d7/migrator/0.log"
Sep 05 03:09:58.982284 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:09:58.982259    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-s54l4_f68966fc-2067-46e7-a102-aa8af80ad1d7/graceful-termination/0.log"
Sep 05 03:10:00.345882 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:10:00.345753    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6wbh8_5fa3cda0-ec69-4fe0-b459-119a5d7d4d42/kube-multus/0.log"
Sep 05 03:10:00.372240 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:10:00.372212    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-279ft_4b864c4d-5153-4fed-a470-e6848614f613/kube-multus-additional-cni-plugins/0.log"
Sep 05 03:10:00.396771 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:10:00.396743    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-279ft_4b864c4d-5153-4fed-a470-e6848614f613/egress-router-binary-copy/0.log"
Sep 05 03:10:00.421808 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:10:00.421782    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-279ft_4b864c4d-5153-4fed-a470-e6848614f613/cni-plugins/0.log"
Sep 05 03:10:00.446136 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:10:00.446094    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-279ft_4b864c4d-5153-4fed-a470-e6848614f613/bond-cni-plugin/0.log"
Sep 05 03:10:00.468515 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:10:00.468481    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-279ft_4b864c4d-5153-4fed-a470-e6848614f613/routeoverride-cni/0.log"
Sep 05 03:10:00.491385 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:10:00.491356    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-279ft_4b864c4d-5153-4fed-a470-e6848614f613/whereabouts-cni-bincopy/0.log"
Sep 05 03:10:00.515313 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:10:00.515280    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-279ft_4b864c4d-5153-4fed-a470-e6848614f613/whereabouts-cni/0.log"
Sep 05 03:10:01.027436 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:10:01.027382    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-262h2_e701e309-b8b1-47c5-927a-ce15959fafa1/network-metrics-daemon/0.log"
Sep 05 03:10:01.047317 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:10:01.047287    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-262h2_e701e309-b8b1-47c5-927a-ce15959fafa1/kube-rbac-proxy/0.log"
Sep 05 03:10:01.890962 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:10:01.890930    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7vnhf_f16c6647-7313-4ab5-b757-d95b95979d3c/ovn-controller/0.log"
Sep 05 03:10:01.908136 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:10:01.908108    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7vnhf_f16c6647-7313-4ab5-b757-d95b95979d3c/ovn-acl-logging/0.log"
Sep 05 03:10:01.938627 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:10:01.938590    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7vnhf_f16c6647-7313-4ab5-b757-d95b95979d3c/kube-rbac-proxy-node/0.log"
Sep 05 03:10:01.964119 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:10:01.964091    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7vnhf_f16c6647-7313-4ab5-b757-d95b95979d3c/kube-rbac-proxy-ovn-metrics/0.log"
Sep 05 03:10:01.988010 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:10:01.987977    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7vnhf_f16c6647-7313-4ab5-b757-d95b95979d3c/northd/0.log"
Sep 05 03:10:02.010760 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:10:02.010730    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7vnhf_f16c6647-7313-4ab5-b757-d95b95979d3c/nbdb/0.log"
Sep 05 03:10:02.034640 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:10:02.034607    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7vnhf_f16c6647-7313-4ab5-b757-d95b95979d3c/sbdb/0.log"
Sep 05 03:10:02.095483 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:10:02.095451    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7vnhf_f16c6647-7313-4ab5-b757-d95b95979d3c/ovnkube-controller/0.log"
Sep 05 03:10:03.511521 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:10:03.511488    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-55qfp_ac1f1f3d-3021-4a29-9162-77a512ca3a29/network-check-target-container/0.log"
Sep 05 03:10:04.467500 ip-10-0-130-182 kubenswrapper[2566]: I0905 03:10:04.467472    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-pp29d_7e81e0ea-1618-456d-b0f4-33c3e3f3e518/iptables-alerter/0.log"