Jul 27 21:01:49.508359 ip-10-0-138-50 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 27 21:01:49.508373 ip-10-0-138-50 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 27 21:01:49.508382 ip-10-0-138-50 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 27 21:01:49.508698 ip-10-0-138-50 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 27 21:01:59.553685 ip-10-0-138-50 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 27 21:01:59.553705 ip-10-0-138-50 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 623a549b33e347d584cd1483ff8fcb30 --
Jul 27 21:03:39.954178 ip-10-0-138-50 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 21:03:40.316079 ip-10-0-138-50 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 21:03:40.316079 ip-10-0-138-50 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 21:03:40.316079 ip-10-0-138-50 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 21:03:40.316079 ip-10-0-138-50 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 21:03:40.316079 ip-10-0-138-50 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 21:03:40.317772 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.317661    2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 21:03:40.322183 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322169    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:03:40.322183 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322183    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:03:40.322246 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322187    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:03:40.322246 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322190    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:03:40.322246 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322195    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:03:40.322246 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322198    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:03:40.322246 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322201    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:03:40.322246 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322210    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:03:40.322246 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322213    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:03:40.322246 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322216    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:03:40.322246 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322219    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:03:40.322246 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322222    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:03:40.322246 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322225    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:03:40.322246 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322227    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:03:40.322246 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322230    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:03:40.322246 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322232    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:03:40.322246 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322235    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:03:40.322246 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322237    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:03:40.322246 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322241    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:03:40.322246 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322245    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:03:40.322246 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322248    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:03:40.322246 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322251    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:03:40.322719 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322254    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:03:40.322719 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322257    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:03:40.322719 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322260    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:03:40.322719 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322263    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:03:40.322719 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322266    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:03:40.322719 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322268    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:03:40.322719 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322271    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:03:40.322719 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322273    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:03:40.322719 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322276    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:03:40.322719 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322278    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:03:40.322719 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322281    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:03:40.322719 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322283    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:03:40.322719 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322286    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:03:40.322719 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322288    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:03:40.322719 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322291    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:03:40.322719 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322293    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:03:40.322719 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322296    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:03:40.322719 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322299    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:03:40.322719 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322302    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:03:40.322719 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322304    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:03:40.323250 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322307    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:03:40.323250 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322309    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:03:40.323250 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322312    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:03:40.323250 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322314    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:03:40.323250 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322316    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:03:40.323250 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322319    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:03:40.323250 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322321    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:03:40.323250 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322324    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:03:40.323250 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322326    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:03:40.323250 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322328    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:03:40.323250 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322331    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:03:40.323250 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322333    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:03:40.323250 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322336    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:03:40.323250 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322339    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:03:40.323250 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322341    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:03:40.323250 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322344    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:03:40.323250 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322347    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:03:40.323250 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322349    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:03:40.323250 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322352    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:03:40.323250 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322354    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:03:40.323753 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322357    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:03:40.323753 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322359    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:03:40.323753 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322362    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:03:40.323753 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322364    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:03:40.323753 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322367    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:03:40.323753 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322370    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:03:40.323753 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322372    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:03:40.323753 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322375    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:03:40.323753 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322377    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:03:40.323753 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322380    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:03:40.323753 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322382    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:03:40.323753 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322385    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:03:40.323753 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322387    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:03:40.323753 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322390    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:03:40.323753 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322392    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:03:40.323753 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322396    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:03:40.323753 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322398    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:03:40.323753 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322401    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:03:40.323753 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322403    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:03:40.323753 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322405    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:03:40.324441 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322408    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:03:40.324441 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322410    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:03:40.324441 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322413    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:03:40.324441 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322416    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:03:40.324441 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322419    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:03:40.324441 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322423    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:03:40.324441 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322427    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:03:40.324441 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322429    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:03:40.324441 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322432    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:03:40.324441 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322435    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:03:40.324441 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322438    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:03:40.324441 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322440    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:03:40.324441 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322443    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:03:40.324441 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322445    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:03:40.324441 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322448    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:03:40.324441 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322450    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:03:40.324441 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.322454    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:03:40.324441 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323153    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:03:40.324975 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323162    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:03:40.324975 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323166    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:03:40.324975 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323170    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:03:40.324975 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323174    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:03:40.324975 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323177    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:03:40.324975 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323180    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:03:40.324975 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323184    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:03:40.324975 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323187    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:03:40.324975 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323189    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:03:40.324975 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323192    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:03:40.324975 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323195    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:03:40.324975 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323198    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:03:40.324975 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323200    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:03:40.324975 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323203    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:03:40.324975 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323205    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:03:40.324975 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323208    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:03:40.324975 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323210    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:03:40.324975 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323214    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:03:40.324975 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323217    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:03:40.325460 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323220    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:03:40.325460 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323222    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:03:40.325460 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323225    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:03:40.325460 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323227    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:03:40.325460 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323230    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:03:40.325460 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323232    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:03:40.325460 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323235    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:03:40.325460 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323238    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:03:40.325460 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323240    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:03:40.325460 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323243    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:03:40.325460 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323245    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:03:40.325460 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323248    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:03:40.325460 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323250    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:03:40.325460 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323253    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:03:40.325460 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323255    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:03:40.325460 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323258    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:03:40.325460 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323261    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:03:40.325460 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323263    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:03:40.325460 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323266    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:03:40.325460 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323268    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:03:40.325945 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323271    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:03:40.325945 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323274    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:03:40.325945 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323276    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:03:40.325945 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323279    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:03:40.325945 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323281    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:03:40.325945 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323284    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:03:40.325945 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323286    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:03:40.325945 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323289    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:03:40.325945 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323291    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:03:40.325945 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323294    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:03:40.325945 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323296    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:03:40.325945 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323299    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:03:40.325945 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323302    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:03:40.325945 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323305    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:03:40.325945 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323308    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:03:40.325945 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323310    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:03:40.325945 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323313    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:03:40.325945 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323315    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:03:40.325945 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323318    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:03:40.325945 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323320    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:03:40.326481 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323323    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:03:40.326481 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323325    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:03:40.326481 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323328    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:03:40.326481 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323331    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:03:40.326481 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323333    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:03:40.326481 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323336    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:03:40.326481 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323338    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:03:40.326481 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323340    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:03:40.326481 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323343    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:03:40.326481 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323346    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:03:40.326481 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323348    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:03:40.326481 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323351    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:03:40.326481 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323353    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:03:40.326481 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323356    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:03:40.326481 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323359    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:03:40.326481 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323361    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:03:40.326481 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323364    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:03:40.326481 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323366    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:03:40.326481 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323369    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:03:40.326941 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323373    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:03:40.326941 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323376    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:03:40.326941 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323378    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:03:40.326941 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323381    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:03:40.326941 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323383    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:03:40.326941 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323386    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:03:40.326941 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323388    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:03:40.326941 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323391    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:03:40.326941 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323393    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:03:40.326941 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323396    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:03:40.326941 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323398    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:03:40.326941 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323401    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:03:40.326941 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323403    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:03:40.326941 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323406    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:03:40.326941 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323408    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:03:40.326941 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323411    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:03:40.326941 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323413    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:03:40.326941 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323417    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:03:40.326941 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323420    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:03:40.326941 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.323423    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:03:40.327455 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326117    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:03:40.327455 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326129    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:03:40.327455 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326132    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:03:40.327455 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326135    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:03:40.327455 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326138    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:03:40.327455 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326140    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:03:40.327455 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326143    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:03:40.327455 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326145    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:03:40.327455 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326150    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:03:40.327455 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326153    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:03:40.327455 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326156    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:03:40.327455 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326159    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:03:40.327455 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326163    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:03:40.327455 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326165    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:03:40.327455 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326168    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:03:40.327455 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326170    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:03:40.327455 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326173    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:03:40.327455 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326177    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:03:40.327455 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326181    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:03:40.327917 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326184    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:03:40.327917 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326186    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:03:40.327917 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326189    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:03:40.327917 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326192    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:03:40.327917 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326195    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:03:40.327917 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326198    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:03:40.327917 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326201    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:03:40.327917 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326203    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:03:40.327917 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326206    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:03:40.327917 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326208    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:03:40.327917 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326211    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:03:40.327917 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326213    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:03:40.327917 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326216    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:03:40.327917 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326219    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:03:40.327917 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326222    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:03:40.327917 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326225    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:03:40.327917 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326227    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:03:40.327917 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326230    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:03:40.327917 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326232    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:03:40.327917 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326235    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:03:40.328434 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326239    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:03:40.328434 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326243    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:03:40.328434 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326245    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:03:40.328434 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326248    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:03:40.328434 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326252    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:03:40.328434 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326254    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:03:40.328434 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326257    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:03:40.328434 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326259    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:03:40.328434 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326261    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:03:40.328434 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326264    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:03:40.328434 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326267    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:03:40.328434 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326270    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:03:40.328434 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326272    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:03:40.328434 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326275    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:03:40.328434 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326277    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:03:40.328434 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326280    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:03:40.328434 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326283    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:03:40.328434 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326287    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:03:40.328434 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326291    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:03:40.328434 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326295    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:03:40.328943 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326299    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:03:40.328943 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326303    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:03:40.328943 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326306    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:03:40.328943 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326310    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:03:40.328943 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326314    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:03:40.328943 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326319    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:03:40.328943 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326322    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:03:40.328943 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326325    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:03:40.328943 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326329    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:03:40.328943 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326331    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:03:40.328943 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326334    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:03:40.328943 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326336    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:03:40.328943 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326339    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:03:40.328943 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326341    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:03:40.328943 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326344    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:03:40.328943 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326346    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:03:40.328943 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326349    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:03:40.328943 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326351    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:03:40.328943 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326354    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:03:40.328943 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326356    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:03:40.329451 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326359    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:03:40.329451 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326361    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:03:40.329451 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326363    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:03:40.329451 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326366    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:03:40.329451 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326371    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:03:40.329451 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326374    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:03:40.329451 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326377    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:03:40.329451 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326379    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:03:40.329451 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326382    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:03:40.329451 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326384    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:03:40.329451 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326387    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:03:40.329451 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326389    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:03:40.329451 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326392    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:03:40.329451 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326394    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:03:40.329451 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326397    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:03:40.329451 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326399    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:03:40.329451 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326401    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:03:40.329451 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326404    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:03:40.329451 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326407    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.326410    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326484    2571 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326491    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326499    2571 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326503    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326508    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326511    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326515    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326520    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326523    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326526    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326529    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326533    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326536    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326539    2571 flags.go:64] FLAG: --cgroup-root=""
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326541    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326544    2571 flags.go:64] FLAG: --client-ca-file=""
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326547    2571 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326550    2571 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326555    2571 flags.go:64] FLAG: --cluster-domain=""
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326558    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326561    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326564    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 21:03:40.329908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326567    2571 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326571    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326574    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326577    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326580    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326583    2571 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326586    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326589    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326593    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326596    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326600    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326603    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326606    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326609    2571 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326611    2571 flags.go:64] FLAG: --enable-server="true"
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326614    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326618    2571 flags.go:64] FLAG: --event-burst="100"
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326622    2571 flags.go:64] FLAG: --event-qps="50"
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326624    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326627    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326630    2571 flags.go:64] FLAG: --eviction-hard=""
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326633    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326636    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326639    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326641    2571 flags.go:64] FLAG: --eviction-soft=""
Jul 27 21:03:40.330510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326644    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326647    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326650    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326653    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326656    2571 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326659    2571 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326661    2571 flags.go:64] FLAG: --feature-gates=""
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326665    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326668    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326671    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326674    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326676    2571 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326679    2571 flags.go:64] FLAG: --help="false"
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326682    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-138-50.ec2.internal"
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326685    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326688    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326691    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326695    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326698    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326701    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326703    2571 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326706    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326709    2571 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 21:03:40.331149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326711    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326714    2571 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326717    2571 flags.go:64] FLAG: --kube-reserved=""
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326720    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326722    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326725    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326728    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326731    2571 flags.go:64] FLAG: --lock-file=""
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326733    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326736    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326739    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326745    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326747    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326751    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326753    2571 flags.go:64] FLAG: --logging-format="text"
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326756    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326759    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326762    2571 flags.go:64] FLAG: --manifest-url=""
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326765    2571 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326769    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326772    2571 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326776    2571 flags.go:64] FLAG: --max-pods="110"
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326780    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326783    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326786    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 21:03:40.331880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326789    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326791    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326794    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326797    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326805    2571 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326808    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326811    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326814    2571 flags.go:64] FLAG: --pod-cidr=""
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326816    2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326821    2571 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326825    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326828    2571 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326830    2571 flags.go:64] FLAG: --port="10250"
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326833    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326836    2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-025ae262b0cf46ba4"
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326839    2571 flags.go:64] FLAG: --qos-reserved=""
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326842    2571 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326845    2571 flags.go:64] FLAG: --register-node="true"
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326847    2571 flags.go:64] FLAG: --register-with-taints=""
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326851    2571 flags.go:64] FLAG: --registry-burst="10"
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326853    2571 flags.go:64] FLAG: --registry-qps="5"
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326858    2571 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326860    2571 flags.go:64] FLAG: --reserved-memory=""
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326864    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326867    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 21:03:40.332508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326870    2571 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326872    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326875    2571 flags.go:64] FLAG: --runonce="false"
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326878    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326880    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326883    2571 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326886    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326888    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326891    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326894    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326897    2571 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326899    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326902    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326905    2571 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326907    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326910    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326913    2571 flags.go:64] FLAG: --system-cgroups=""
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326916    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326921    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326923    2571 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326926    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326930    2571 flags.go:64] FLAG: --tls-min-version=""
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326933    2571 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326935    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326938    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 21:03:40.333162 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326940    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 21:03:40.333763 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326943    2571 flags.go:64] FLAG: --v="2"
Jul 27 21:03:40.333763 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326947    2571 flags.go:64] FLAG: --version="false"
Jul 27 21:03:40.333763 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326951    2571 flags.go:64] FLAG: --vmodule=""
Jul 27 21:03:40.333763 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326971    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 21:03:40.333763 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.326977    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 21:03:40.333763 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327087    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:03:40.333763 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327091    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:03:40.333763 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327094    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:03:40.333763 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327097    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:03:40.333763 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327099    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:03:40.333763 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327103    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:03:40.333763 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327106    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:03:40.333763 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327109    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:03:40.333763 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327113    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:03:40.333763 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327116    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:03:40.333763 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327120    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:03:40.333763 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327122    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:03:40.333763 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327125    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:03:40.333763 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327127    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:03:40.333763 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327130    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:03:40.333763 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327132    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:03:40.334316 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327135    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:03:40.334316 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327138    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:03:40.334316 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327140    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:03:40.334316 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327143    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:03:40.334316 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327145    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:03:40.334316 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327147    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:03:40.334316 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327150    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:03:40.334316 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327153    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:03:40.334316 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327155    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:03:40.334316 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327158    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:03:40.334316 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327160    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:03:40.334316 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327163    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:03:40.334316 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327165    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:03:40.334316 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327168    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:03:40.334316 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327170    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:03:40.334316 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327175    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:03:40.334316 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327177    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:03:40.334316 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327180    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:03:40.334316 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327182    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:03:40.334316 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327185    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:03:40.334817 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327187    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:03:40.334817 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327191    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:03:40.334817 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327193    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:03:40.334817 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327196    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:03:40.334817 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327198    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:03:40.334817 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327201    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:03:40.334817 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327203    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:03:40.334817 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327206    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:03:40.334817 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327208    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:03:40.334817 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327211    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:03:40.334817 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327213    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:03:40.334817 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327216    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:03:40.334817 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327218    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:03:40.334817 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327221    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:03:40.334817 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327223    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:03:40.334817 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327226    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:03:40.334817 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327228    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:03:40.334817 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327230    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:03:40.334817 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327233    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:03:40.334817 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327235    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:03:40.335362 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327237    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:03:40.335362 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327240    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:03:40.335362 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327242    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:03:40.335362 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327245    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:03:40.335362 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327247    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:03:40.335362 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327250    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:03:40.335362 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327252    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:03:40.335362 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327257    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:03:40.335362 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327261    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:03:40.335362 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327263    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:03:40.335362 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327266    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:03:40.335362 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327268    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:03:40.335362 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327271    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:03:40.335362 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327275    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:03:40.335362 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327277    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:03:40.335362 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327280    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:03:40.335362 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327282    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:03:40.335362 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327284    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:03:40.335362 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327287    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:03:40.335362 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327290    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:03:40.335846 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327293    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:03:40.335846 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327295    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:03:40.335846 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327299    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:03:40.335846 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327302    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:03:40.335846 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327305    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:03:40.335846 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327308    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:03:40.335846 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327311    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:03:40.335846 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327314    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:03:40.335846 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327316    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:03:40.335846 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327319    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:03:40.335846 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327322    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:03:40.335846 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327326    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:03:40.335846 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327329    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:03:40.335846 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327331    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:03:40.335846 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327334    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:03:40.335846 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327337    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:03:40.335846 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327340    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:03:40.335846 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327342    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:03:40.335846 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327345    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:03:40.336318 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327349    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:03:40.336318 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327351    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:03:40.336318 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327355    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:03:40.336318 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.327358    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:03:40.336318 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.327999    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 21:03:40.336318 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.335053    2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 21:03:40.336318 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.335067    2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 21:03:40.336318 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.335088    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 21:03:40.336318 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.335094    2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 21:03:40.336318 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335152    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:03:40.336318 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335157    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:03:40.336318 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335160    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:03:40.336318 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335164    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:03:40.336318 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335167    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:03:40.336318 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335169    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:03:40.336703 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335172    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:03:40.336703 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335174    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:03:40.336703 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335177    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:03:40.336703 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335179    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:03:40.336703 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335181    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:03:40.336703 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335184    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:03:40.336703 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335187    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:03:40.336703 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335189    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:03:40.336703 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335191    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:03:40.336703 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335195    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:03:40.336703 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335199    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:03:40.336703 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335202    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:03:40.336703 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335205    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:03:40.336703 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335207    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:03:40.336703 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335210    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:03:40.336703 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335212    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:03:40.336703 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335215    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:03:40.336703 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335218    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:03:40.336703 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335220    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:03:40.336703 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335223    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:03:40.337212 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335225    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:03:40.337212 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335228    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:03:40.337212 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335230    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:03:40.337212 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335234    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:03:40.337212 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335237    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:03:40.337212 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335239    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:03:40.337212 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335242    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:03:40.337212 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335245    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:03:40.337212 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335247    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:03:40.337212 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335250    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:03:40.337212 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335252    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:03:40.337212 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335255    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:03:40.337212 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335257    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:03:40.337212 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335260    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:03:40.337212 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335262    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:03:40.337212 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335265    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:03:40.337212 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335267    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:03:40.337212 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335270    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:03:40.337212 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335272    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:03:40.337678 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335275    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:03:40.337678 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335277    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:03:40.337678 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335280    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:03:40.337678 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335283    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:03:40.337678 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335286    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:03:40.337678 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335288    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:03:40.337678 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335290    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:03:40.337678 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335293    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:03:40.337678 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335295    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:03:40.337678 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335297    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:03:40.337678 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335300    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:03:40.337678 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335302    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:03:40.337678 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335305    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:03:40.337678 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335307    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:03:40.337678 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335309    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:03:40.337678 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335312    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:03:40.337678 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335315    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:03:40.337678 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335318    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:03:40.337678 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335321    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:03:40.337678 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335323    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:03:40.338186 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335326    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:03:40.338186 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335328    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:03:40.338186 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335331    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:03:40.338186 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335333    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:03:40.338186 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335336    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:03:40.338186 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335338    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:03:40.338186 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335341    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:03:40.338186 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335343    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:03:40.338186 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335346    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:03:40.338186 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335348    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:03:40.338186 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335351    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:03:40.338186 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335353    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:03:40.338186 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335357    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:03:40.338186 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335361    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:03:40.338186 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335365    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:03:40.338186 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335368    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:03:40.338186 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335371    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:03:40.338186 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335373    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:03:40.338186 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335376    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:03:40.338655 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335379    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:03:40.338655 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335381    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:03:40.338655 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335384    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:03:40.338655 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335386    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:03:40.338655 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335389    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:03:40.338655 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335392    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:03:40.338655 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335394    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:03:40.338655 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335397    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:03:40.338655 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335400    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:03:40.338655 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335403    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:03:40.338655 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335406    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:03:40.338655 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335408    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:03:40.338655 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335411    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:03:40.338655 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335413    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:03:40.338655 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335416    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:03:40.339061 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.335421    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 21:03:40.339061 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335543    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:03:40.339061 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335548    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:03:40.339061 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335551    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:03:40.339061 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335553    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:03:40.339061 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335556    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:03:40.339061 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335559    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:03:40.339061 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335561    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:03:40.339061 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335564    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:03:40.339061 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335566    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:03:40.339061 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335569    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:03:40.339061 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335571    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:03:40.339061 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335574    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:03:40.339061 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335576    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:03:40.339061 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335579    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:03:40.339436 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335581    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:03:40.339436 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335585    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:03:40.339436 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335589    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:03:40.339436 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335592    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:03:40.339436 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335594    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:03:40.339436 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335597    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:03:40.339436 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335600    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:03:40.339436 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335602    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:03:40.339436 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335605    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:03:40.339436 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335607    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:03:40.339436 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335610    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:03:40.339436 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335612    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:03:40.339436 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335615    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:03:40.339436 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335617    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:03:40.339436 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335620    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:03:40.339436 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335623    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:03:40.339436 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335625    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:03:40.339436 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335628    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:03:40.339436 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335630    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:03:40.339897 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335632    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:03:40.339897 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335635    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:03:40.339897 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335637    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:03:40.339897 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335640    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:03:40.339897 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335642    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:03:40.339897 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335644    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:03:40.339897 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335647    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:03:40.339897 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335649    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:03:40.339897 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335651    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:03:40.339897 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335654    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:03:40.339897 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335656    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:03:40.339897 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335659    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:03:40.339897 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335661    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:03:40.339897 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335665    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:03:40.339897 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335668    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:03:40.339897 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335671    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:03:40.339897 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335673    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:03:40.339897 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335676    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:03:40.339897 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335678    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:03:40.339897 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335681    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:03:40.340390 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335683    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:03:40.340390 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335685    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:03:40.340390 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335688    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:03:40.340390 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335690    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:03:40.340390 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335693    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:03:40.340390 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335696    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:03:40.340390 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335698    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:03:40.340390 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335701    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:03:40.340390 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335704    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:03:40.340390 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335706    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:03:40.340390 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335708    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:03:40.340390 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335711    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:03:40.340390 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335714    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:03:40.340390 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335717    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:03:40.340390 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335720    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:03:40.340390 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335723    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:03:40.340390 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335726    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:03:40.340390 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335729    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:03:40.340390 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335732    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:03:40.340857 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335734    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:03:40.340857 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335737    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:03:40.340857 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335739    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:03:40.340857 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335742    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:03:40.340857 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335744    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:03:40.340857 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335747    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:03:40.340857 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335749    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:03:40.340857 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335752    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:03:40.340857 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335754    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:03:40.340857 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335757    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:03:40.340857 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335759    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:03:40.340857 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335762    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:03:40.340857 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335765    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:03:40.340857 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335767    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:03:40.340857 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335770    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:03:40.340857 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335772    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:03:40.340857 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335775    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:03:40.340857 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335778    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:03:40.340857 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335781    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:03:40.340857 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335783    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:03:40.341352 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335786    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:03:40.341352 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335788    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:03:40.341352 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335791    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:03:40.341352 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335794    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:03:40.341352 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335796    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:03:40.341352 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335799    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:03:40.341352 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:40.335801    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:03:40.341352 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.335805    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 21:03:40.341352 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.335901    2571 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 21:03:40.341352 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.338171    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 21:03:40.341352 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.339462    2571 server.go:1019] "Starting client certificate rotation"
Jul 27 21:03:40.341352 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.339563    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 21:03:40.341352 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.339590    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 21:03:40.342093 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.342020    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 21:03:40.343454 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.343440    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 21:03:40.353497 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.353481    2571 log.go:25] "Validated CRI v1 runtime API"
Jul 27 21:03:40.356146 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.356128    2571 log.go:25] "Validated CRI v1 image API"
Jul 27 21:03:40.357470 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.357443    2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 21:03:40.360012 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.359993    2571 fs.go:135] Filesystem UUIDs: map[410e7221-9f85-4da5-9370-f82ffbf8e0a7:/dev/nvme0n1p3 44f2e14e-1dc3-4f83-a222-f0b9f4b0e88c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 27 21:03:40.360092 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.360010    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 21:03:40.365705 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.365595    2571 manager.go:216] Machine: {Timestamp:2026-07-27 21:03:40.363636186 +0000 UTC m=+0.298852556 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103217 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29e72362510ac5eee668264fc116c0 SystemUUID:ec29e723-6251-0ac5-eee6-68264fc116c0 BootID:623a549b-33e3-47d5-84cd-1483ff8fcb30 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:91:27:d2:b9:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:91:27:d2:b9:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:45:1c:66:ee:be Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 21:03:40.365705 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.365703    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 21:03:40.365826 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.365814    2571 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 21:03:40.368731 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.368703    2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 21:03:40.368875 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.368732    2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-50.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 21:03:40.369107 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.369089    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 21:03:40.369435 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.369423    2571 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 21:03:40.369483 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.369437    2571 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 21:03:40.369483 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.369445    2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 21:03:40.370265 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.370255    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 21:03:40.371715 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.371705    2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 21:03:40.374815 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.374804    2571 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 21:03:40.374940 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.374931    2571 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 21:03:40.375004 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.374977    2571 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 21:03:40.375004 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.374999    2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 21:03:40.375101 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.375018    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 21:03:40.375101 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.375028    2571 kubelet.go:398] "Adding apiserver pod source"
Jul 27 21:03:40.375101 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.375036    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 21:03:40.375391 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.375379    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 21:03:40.375447 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.375397    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 21:03:40.378006 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.377992    2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 21:03:40.380161 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.380149    2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 21:03:40.380202 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.380179    2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 21:03:40.381815 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.381802    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 21:03:40.381848 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.381824    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 21:03:40.381848 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.381834    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 21:03:40.381848 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.381843    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 21:03:40.381932 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.381856    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 21:03:40.381932 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.381865    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 21:03:40.381932 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.381877    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 21:03:40.381932 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.381886    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 21:03:40.382047 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.381936    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 21:03:40.382047 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.381947    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 21:03:40.382047 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.381978    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 21:03:40.382047 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.381987    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 21:03:40.382722 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.382711    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 21:03:40.382754 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.382724    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 21:03:40.385880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.385866    2571 server.go:1268] "Started kubelet"
Jul 27 21:03:40.386031 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.386005    2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 21:03:40.386165 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.385986    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 21:03:40.386232 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.386218    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 21:03:40.386734 ip-10-0-138-50 systemd[1]: Started Kubernetes Kubelet.
Jul 27 21:03:40.387555 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.387541    2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 21:03:40.388212 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.388191    2571 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 21:03:40.390360 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.390339    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-50.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 21:03:40.390893 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:40.390847    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-50.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 21:03:40.391000 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:40.390915    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 21:03:40.392608 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:40.392569    2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 21:03:40.393323 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:40.392453    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-50.ec2.internal.18c64168bc051bec  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-50.ec2.internal,UID:ip-10-0-138-50.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-50.ec2.internal,},FirstTimestamp:2026-07-27 21:03:40.385836012 +0000 UTC m=+0.321052382,LastTimestamp:2026-07-27 21:03:40.385836012 +0000 UTC m=+0.321052382,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-50.ec2.internal,}"
Jul 27 21:03:40.393663 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.393648    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jht8d"
Jul 27 21:03:40.395658 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.395641    2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 21:03:40.395754 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:40.395735    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-50.ec2.internal\" not found"
Jul 27 21:03:40.395810 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.395645    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 21:03:40.396278 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.396258    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 21:03:40.396367 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.396302    2571 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 21:03:40.396367 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.396313    2571 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 21:03:40.396483 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.396446    2571 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 21:03:40.396483 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.396459    2571 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 21:03:40.396673 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.396659    2571 factory.go:55] Registering systemd factory
Jul 27 21:03:40.396739 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.396676    2571 factory.go:223] Registration of the systemd container factory successfully
Jul 27 21:03:40.396849 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.396837    2571 factory.go:153] Registering CRI-O factory
Jul 27 21:03:40.396889 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.396852    2571 factory.go:223] Registration of the crio container factory successfully
Jul 27 21:03:40.396917 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.396890    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 21:03:40.396917 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.396907    2571 factory.go:103] Registering Raw factory
Jul 27 21:03:40.397035 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.396920    2571 manager.go:1196] Started watching for new ooms in manager
Jul 27 21:03:40.397293 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.397282    2571 manager.go:349] Starting recovery of all containers
Jul 27 21:03:40.401408 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.401305    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jht8d"
Jul 27 21:03:40.402011 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:40.401989    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 21:03:40.402104 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:40.402044    2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-50.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 21:03:40.407092 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.407072    2571 manager.go:354] Recovery completed
Jul 27 21:03:40.411568 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.411551    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:03:40.414177 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.414156    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:03:40.414221 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.414190    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:03:40.414221 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.414202    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-50.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:03:40.414705 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.414691    2571 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 21:03:40.414705 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.414698    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:03:40.414826 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.414702    2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 21:03:40.414826 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.414763    2571 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 21:03:40.415450 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.415433    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:03:40.415517 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.415463    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:03:40.415517 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.415475    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-50.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:03:40.416480 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.416468    2571 policy_none.go:49] "None policy: Start"
Jul 27 21:03:40.416540 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.416484    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 21:03:40.416540 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.416493    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 21:03:40.416726 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:40.416668    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-50.ec2.internal.18c64168bdb5778f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-50.ec2.internal,UID:ip-10-0-138-50.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-50.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-50.ec2.internal,},FirstTimestamp:2026-07-27 21:03:40.414171023 +0000 UTC m=+0.349387398,LastTimestamp:2026-07-27 21:03:40.414171023 +0000 UTC m=+0.349387398,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-50.ec2.internal,}"
Jul 27 21:03:40.417868 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.417853    2571 policy_none.go:47] "Start"
Jul 27 21:03:40.451239 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.451129    2571 manager.go:337] "Starting Device Plugin manager"
Jul 27 21:03:40.451339 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:40.451259    2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 21:03:40.451339 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.451272    2571 server.go:87] "Starting device plugin registration server"
Jul 27 21:03:40.451505 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.451493    2571 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 21:03:40.451570 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.451507    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 21:03:40.451613 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.451597    2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 21:03:40.451691 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.451668    2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 21:03:40.451691 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.451680    2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 21:03:40.452211 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:40.452190    2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 21:03:40.452264 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:40.452245    2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-50.ec2.internal\" not found"
Jul 27 21:03:40.551663 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.551632    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:03:40.552916 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.552703    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:03:40.552916 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.552732    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:03:40.552916 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.552744    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-50.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:03:40.552916 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.552779    2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-50.ec2.internal"
Jul 27 21:03:40.560551 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.560536    2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-50.ec2.internal"
Jul 27 21:03:40.560616 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:40.560558    2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-50.ec2.internal\": node \"ip-10-0-138-50.ec2.internal\" not found"
Jul 27 21:03:40.565651 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.565630    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 21:03:40.566928 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.566877    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 21:03:40.566928 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.566911    2571 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 21:03:40.566928 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.566929    2571 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 21:03:40.567111 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:40.566983    2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 21:03:40.569243 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.569221    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:03:40.576204 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:40.576188    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-50.ec2.internal\" not found"
Jul 27 21:03:40.667404 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.667364    2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-50.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-50.ec2.internal"]
Jul 27 21:03:40.667575 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.667452    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:03:40.670488 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.670467    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:03:40.670611 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.670501    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:03:40.670611 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.670511    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-50.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:03:40.671893 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.671881    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:03:40.672012 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.671996    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-50.ec2.internal"
Jul 27 21:03:40.672063 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.672038    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:03:40.672668 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.672649    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:03:40.672774 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.672683    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:03:40.672774 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.672693    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:03:40.672774 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.672716    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:03:40.672774 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.672730    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-50.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:03:40.672774 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.672696    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-50.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:03:40.674129 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.674111    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-50.ec2.internal"
Jul 27 21:03:40.674216 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.674141    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:03:40.675057 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.675035    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:03:40.675146 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.675061    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:03:40.675146 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.675071    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-50.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:03:40.676973 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:40.676945    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-50.ec2.internal\" not found"
Jul 27 21:03:40.698584 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.698561    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fda18b08661970af357d9091388de747-config\") pod \"kube-apiserver-proxy-ip-10-0-138-50.ec2.internal\" (UID: \"fda18b08661970af357d9091388de747\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-50.ec2.internal"
Jul 27 21:03:40.698700 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.698590    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ad061777f914723d1969e77779e70368-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-50.ec2.internal\" (UID: \"ad061777f914723d1969e77779e70368\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-50.ec2.internal"
Jul 27 21:03:40.698700 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.698613    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad061777f914723d1969e77779e70368-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-50.ec2.internal\" (UID: \"ad061777f914723d1969e77779e70368\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-50.ec2.internal"
Jul 27 21:03:40.701592 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:40.701576    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-50.ec2.internal\" not found" node="ip-10-0-138-50.ec2.internal"
Jul 27 21:03:40.705987 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:40.705968    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-50.ec2.internal\" not found" node="ip-10-0-138-50.ec2.internal"
Jul 27 21:03:40.778051 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:40.778023    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-50.ec2.internal\" not found"
Jul 27 21:03:40.799466 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.799426    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ad061777f914723d1969e77779e70368-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-50.ec2.internal\" (UID: \"ad061777f914723d1969e77779e70368\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-50.ec2.internal"
Jul 27 21:03:40.799466 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.799472    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad061777f914723d1969e77779e70368-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-50.ec2.internal\" (UID: \"ad061777f914723d1969e77779e70368\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-50.ec2.internal"
Jul 27 21:03:40.799622 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.799504    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fda18b08661970af357d9091388de747-config\") pod \"kube-apiserver-proxy-ip-10-0-138-50.ec2.internal\" (UID: \"fda18b08661970af357d9091388de747\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-50.ec2.internal"
Jul 27 21:03:40.799622 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.799558    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fda18b08661970af357d9091388de747-config\") pod \"kube-apiserver-proxy-ip-10-0-138-50.ec2.internal\" (UID: \"fda18b08661970af357d9091388de747\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-50.ec2.internal"
Jul 27 21:03:40.799622 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.799558    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ad061777f914723d1969e77779e70368-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-50.ec2.internal\" (UID: \"ad061777f914723d1969e77779e70368\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-50.ec2.internal"
Jul 27 21:03:40.799622 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:40.799565    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad061777f914723d1969e77779e70368-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-50.ec2.internal\" (UID: \"ad061777f914723d1969e77779e70368\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-50.ec2.internal"
Jul 27 21:03:40.879011 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:40.878921    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-50.ec2.internal\" not found"
Jul 27 21:03:40.979539 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:40.979500    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-50.ec2.internal\" not found"
Jul 27 21:03:41.004041 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:41.003999    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-50.ec2.internal"
Jul 27 21:03:41.010125 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:41.010095    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-50.ec2.internal"
Jul 27 21:03:41.080263 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:41.080225    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-50.ec2.internal\" not found"
Jul 27 21:03:41.180776 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:41.180716    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-50.ec2.internal\" not found"
Jul 27 21:03:41.281269 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:41.281239    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-50.ec2.internal\" not found"
Jul 27 21:03:41.338822 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:41.338791    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 21:03:41.339319 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:41.338921    2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 21:03:41.382261 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:41.382230    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-50.ec2.internal\" not found"
Jul 27 21:03:41.406278 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:41.406246    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 20:58:40 +0000 UTC" deadline="2028-03-24 01:49:27.033695576 +0000 UTC"
Jul 27 21:03:41.406278 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:41.406273    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14524h45m45.62742637s"
Jul 27 21:03:41.482653 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:41.482610    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-50.ec2.internal\" not found"
Jul 27 21:03:41.560411 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:41.560376    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda18b08661970af357d9091388de747.slice/crio-e51d47a9a8630b0641ac746d0dd87d1f84edfaa3876f7efcfe6c02c1ec19d3fb WatchSource:0}: Error finding container e51d47a9a8630b0641ac746d0dd87d1f84edfaa3876f7efcfe6c02c1ec19d3fb: Status 404 returned error can't find the container with id e51d47a9a8630b0641ac746d0dd87d1f84edfaa3876f7efcfe6c02c1ec19d3fb
Jul 27 21:03:41.560742 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:41.560723    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad061777f914723d1969e77779e70368.slice/crio-5fa9ecc55dedf6a34cf1c26938aef38e8d513bbc90fd0b8bb66042b2d2ed6624 WatchSource:0}: Error finding container 5fa9ecc55dedf6a34cf1c26938aef38e8d513bbc90fd0b8bb66042b2d2ed6624: Status 404 returned error can't find the container with id 5fa9ecc55dedf6a34cf1c26938aef38e8d513bbc90fd0b8bb66042b2d2ed6624
Jul 27 21:03:41.565825 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:41.565810    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:03:41.570543 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:41.570499    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-50.ec2.internal" event={"ID":"fda18b08661970af357d9091388de747","Type":"ContainerStarted","Data":"e51d47a9a8630b0641ac746d0dd87d1f84edfaa3876f7efcfe6c02c1ec19d3fb"}
Jul 27 21:03:41.571583 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:41.571558    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-50.ec2.internal" event={"ID":"ad061777f914723d1969e77779e70368","Type":"ContainerStarted","Data":"5fa9ecc55dedf6a34cf1c26938aef38e8d513bbc90fd0b8bb66042b2d2ed6624"}
Jul 27 21:03:41.583700 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:41.583677    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-50.ec2.internal\" not found"
Jul 27 21:03:41.624446 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:41.624426    2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:03:41.696318 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:41.696295    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-50.ec2.internal"
Jul 27 21:03:41.706826 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:41.706803    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 21:03:41.708498 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:41.708486    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-50.ec2.internal"
Jul 27 21:03:41.714527 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:41.714505    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 21:03:41.738286 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:41.738219    2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:03:41.818056 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:41.818028    2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:03:42.377231 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.377196    2571 apiserver.go:52] "Watching apiserver"
Jul 27 21:03:42.385817 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.385782    2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 21:03:42.389267 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.389227    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-5fg9l","kube-system/kube-apiserver-proxy-ip-10-0-138-50.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt","openshift-dns/node-resolver-gtpdk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-50.ec2.internal","openshift-multus/multus-x8x5c","openshift-multus/network-metrics-daemon-whppm","openshift-cluster-node-tuning-operator/tuned-fb28r","openshift-image-registry/node-ca-cwvnf","openshift-multus/multus-additional-cni-plugins-ttt2t","openshift-network-diagnostics/network-check-target-m2bn8","openshift-network-operator/iptables-alerter-855bl","openshift-ovn-kubernetes/ovnkube-node-xkcbq","kube-system/global-pull-secret-syncer-kvpv2"]
Jul 27 21:03:42.390917 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.390890    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:03:42.391048 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:42.391018    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-whppm" podUID="222ea6ea-61b8-4854-bc88-834449ce2974"
Jul 27 21:03:42.392169 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.392148    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:03:42.392279 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:42.392235    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2bn8" podUID="c268e820-791d-448a-b5ec-83ec69ad08ee"
Jul 27 21:03:42.393353 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.393334    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.394546 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.394523    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gtpdk"
Jul 27 21:03:42.396211 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.395832    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:03:42.396211 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:42.395900    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kvpv2" podUID="71b80e11-ddee-4d86-a0b9-632732d0cca2"
Jul 27 21:03:42.396211 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.395924    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 21:03:42.396211 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.396089    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 21:03:42.396211 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.396097    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 21:03:42.396211 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.396134    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-trrwd\""
Jul 27 21:03:42.396605 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.396585    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 21:03:42.396849 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.396832    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-t9p8x\""
Jul 27 21:03:42.396931 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.396903    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 21:03:42.396993 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.396832    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 21:03:42.397530 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.397511    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.399114 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.399094    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cwvnf"
Jul 27 21:03:42.400026 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.399813    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 21:03:42.400026 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.399860    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 21:03:42.400176 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.400021    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 21:03:42.400231 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.400188    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 21:03:42.400436 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.400415    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gspzp\""
Jul 27 21:03:42.400864 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.400845    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.401514 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.401483    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 21:03:42.401631 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.401590    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 21:03:42.402823 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.402799    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.403015 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.402973    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xc7tp\""
Jul 27 21:03:42.403354 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.403334    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 21:03:42.404055 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.404022    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 21:03:42.404203 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.404185    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 21:03:42.404321 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.404194    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 21:03:42.404321 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.404206    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 21:03:42.404764 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.404747    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.405159 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.405133    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-djnzn\""
Jul 27 21:03:42.405684 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.405667    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b4dmt\""
Jul 27 21:03:42.405785 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.405767    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 21:03:42.405879 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.405861    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:03:42.405954 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.405925    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 21:03:42.406034 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.405978    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 21:03:42.406679 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.406662    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 21:03:42.406786 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.406772    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5fg9l"
Jul 27 21:03:42.407372 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.407355    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 21:03:42.407462 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.407357    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6k78d\""
Jul 27 21:03:42.408249 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.408230    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-855bl"
Jul 27 21:03:42.408741 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.408710    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-host-run-k8s-cni-cncf-io\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.408841 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.408758    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28lxq\" (UniqueName: \"kubernetes.io/projected/72108aac-bf13-4304-8271-b32ad115d0ff-kube-api-access-28lxq\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.408841 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.408812    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/052f9040-58f7-4d2f-9909-bd9014cfd0af-device-dir\") pod \"aws-ebs-csi-driver-node-85vpt\" (UID: \"052f9040-58f7-4d2f-9909-bd9014cfd0af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.408958 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.408840    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71b80e11-ddee-4d86-a0b9-632732d0cca2-dbus\") pod \"global-pull-secret-syncer-kvpv2\" (UID: \"71b80e11-ddee-4d86-a0b9-632732d0cca2\") " pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:03:42.408958 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.408877    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-multus-conf-dir\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.408958 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.408907    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs\") pod \"network-metrics-daemon-whppm\" (UID: \"222ea6ea-61b8-4854-bc88-834449ce2974\") " pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:03:42.408958 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.408931    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9d12f6c-3f55-44b3-b1d5-be1215730520-tmp-dir\") pod \"node-resolver-gtpdk\" (UID: \"c9d12f6c-3f55-44b3-b1d5-be1215730520\") " pod="openshift-dns/node-resolver-gtpdk"
Jul 27 21:03:42.409187 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.408956    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-system-cni-dir\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.409187 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409000    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-multus-cni-dir\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.409187 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409048    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-multus-socket-dir-parent\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.409187 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409075    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-host-run-netns\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.409187 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409097    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/052f9040-58f7-4d2f-9909-bd9014cfd0af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-85vpt\" (UID: \"052f9040-58f7-4d2f-9909-bd9014cfd0af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.409187 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409123    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9d12f6c-3f55-44b3-b1d5-be1215730520-hosts-file\") pod \"node-resolver-gtpdk\" (UID: \"c9d12f6c-3f55-44b3-b1d5-be1215730520\") " pod="openshift-dns/node-resolver-gtpdk"
Jul 27 21:03:42.409187 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409150    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkfgq\" (UniqueName: \"kubernetes.io/projected/c9d12f6c-3f55-44b3-b1d5-be1215730520-kube-api-access-pkfgq\") pod \"node-resolver-gtpdk\" (UID: \"c9d12f6c-3f55-44b3-b1d5-be1215730520\") " pod="openshift-dns/node-resolver-gtpdk"
Jul 27 21:03:42.409187 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409185    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret\") pod \"global-pull-secret-syncer-kvpv2\" (UID: \"71b80e11-ddee-4d86-a0b9-632732d0cca2\") " pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:03:42.409590 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409209    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-host-var-lib-cni-bin\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.409590 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409232    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-host-var-lib-cni-multus\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.409590 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409276    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-hostroot\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.409590 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409338    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-host-run-multus-certs\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.409590 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409370    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-etc-kubernetes\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.409590 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409376    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 21:03:42.409590 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409426    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-host-var-lib-kubelet\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.409590 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409515    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4rgh\" (UniqueName: \"kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh\") pod \"network-check-target-m2bn8\" (UID: \"c268e820-791d-448a-b5ec-83ec69ad08ee\") " pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:03:42.409590 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409546    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/052f9040-58f7-4d2f-9909-bd9014cfd0af-registration-dir\") pod \"aws-ebs-csi-driver-node-85vpt\" (UID: \"052f9040-58f7-4d2f-9909-bd9014cfd0af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.409590 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409587    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71b80e11-ddee-4d86-a0b9-632732d0cca2-kubelet-config\") pod \"global-pull-secret-syncer-kvpv2\" (UID: \"71b80e11-ddee-4d86-a0b9-632732d0cca2\") " pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:03:42.410116 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409612    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-os-release\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.410116 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409640    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9jplt\""
Jul 27 21:03:42.410116 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409658    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72108aac-bf13-4304-8271-b32ad115d0ff-cni-binary-copy\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.410116 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409682    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/052f9040-58f7-4d2f-9909-bd9014cfd0af-socket-dir\") pod \"aws-ebs-csi-driver-node-85vpt\" (UID: \"052f9040-58f7-4d2f-9909-bd9014cfd0af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.410116 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409706    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/052f9040-58f7-4d2f-9909-bd9014cfd0af-etc-selinux\") pod \"aws-ebs-csi-driver-node-85vpt\" (UID: \"052f9040-58f7-4d2f-9909-bd9014cfd0af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.410116 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409730    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-cnibin\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.410116 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409752    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/72108aac-bf13-4304-8271-b32ad115d0ff-multus-daemon-config\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.410116 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409777    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 21:03:42.410116 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409776    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftr86\" (UniqueName: \"kubernetes.io/projected/222ea6ea-61b8-4854-bc88-834449ce2974-kube-api-access-ftr86\") pod \"network-metrics-daemon-whppm\" (UID: \"222ea6ea-61b8-4854-bc88-834449ce2974\") " pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:03:42.410116 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409835    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/052f9040-58f7-4d2f-9909-bd9014cfd0af-sys-fs\") pod \"aws-ebs-csi-driver-node-85vpt\" (UID: \"052f9040-58f7-4d2f-9909-bd9014cfd0af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.410116 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.409866    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct6cp\" (UniqueName: \"kubernetes.io/projected/052f9040-58f7-4d2f-9909-bd9014cfd0af-kube-api-access-ct6cp\") pod \"aws-ebs-csi-driver-node-85vpt\" (UID: \"052f9040-58f7-4d2f-9909-bd9014cfd0af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.410765 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.410258    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 21:03:42.410765 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.410380    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 21:03:42.410765 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.410533    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mvtfq\""
Jul 27 21:03:42.410765 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.410681    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:03:42.412322 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.412289    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 21:03:42.433351 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.433318    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vl2fk"
Jul 27 21:03:42.441463 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.441442    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vl2fk"
Jul 27 21:03:42.497187 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.497144    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 21:03:42.511208 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511167    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/052f9040-58f7-4d2f-9909-bd9014cfd0af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-85vpt\" (UID: \"052f9040-58f7-4d2f-9909-bd9014cfd0af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.511370 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511218    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkfgq\" (UniqueName: \"kubernetes.io/projected/c9d12f6c-3f55-44b3-b1d5-be1215730520-kube-api-access-pkfgq\") pod \"node-resolver-gtpdk\" (UID: \"c9d12f6c-3f55-44b3-b1d5-be1215730520\") " pod="openshift-dns/node-resolver-gtpdk"
Jul 27 21:03:42.511370 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511252    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-env-overrides\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.511370 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511281    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3de75e46-ec68-4801-9493-b1b31f67eefb-host\") pod \"node-ca-cwvnf\" (UID: \"3de75e46-ec68-4801-9493-b1b31f67eefb\") " pod="openshift-image-registry/node-ca-cwvnf"
Jul 27 21:03:42.511370 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511309    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr6gh\" (UniqueName: \"kubernetes.io/projected/3de75e46-ec68-4801-9493-b1b31f67eefb-kube-api-access-kr6gh\") pod \"node-ca-cwvnf\" (UID: \"3de75e46-ec68-4801-9493-b1b31f67eefb\") " pod="openshift-image-registry/node-ca-cwvnf"
Jul 27 21:03:42.511370 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511340    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-host-var-lib-cni-bin\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.511370 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511369    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-host-run-multus-certs\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.511611 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511399    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-etc-kubernetes\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.511611 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511400    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-host-var-lib-cni-bin\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.511611 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511309    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/052f9040-58f7-4d2f-9909-bd9014cfd0af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-85vpt\" (UID: \"052f9040-58f7-4d2f-9909-bd9014cfd0af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.511611 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511425    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-etc-sysctl-d\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.511611 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511459    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-host-run-multus-certs\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.511611 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511462    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-ovnkube-config\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.511611 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511486    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-ovn-node-metrics-cert\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.511611 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511488    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-etc-kubernetes\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.511611 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511504    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8d760a2e-48ae-4021-9766-7c875e7ec1c4-os-release\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.511611 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511524    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-host-var-lib-kubelet\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.511611 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511545    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4rgh\" (UniqueName: \"kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh\") pod \"network-check-target-m2bn8\" (UID: \"c268e820-791d-448a-b5ec-83ec69ad08ee\") " pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:03:42.511611 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511571    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcftf\" (UniqueName: \"kubernetes.io/projected/8d760a2e-48ae-4021-9766-7c875e7ec1c4-kube-api-access-vcftf\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.511611 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511589    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/70daf650-e6e8-4170-83e4-a288b081605e-konnectivity-ca\") pod \"konnectivity-agent-5fg9l\" (UID: \"70daf650-e6e8-4170-83e4-a288b081605e\") " pod="kube-system/konnectivity-agent-5fg9l"
Jul 27 21:03:42.511611 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511602    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-host-var-lib-kubelet\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.512325 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511609    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f2f1d470-36dd-429a-bdc9-191fa2e7b4df-iptables-alerter-script\") pod \"iptables-alerter-855bl\" (UID: \"f2f1d470-36dd-429a-bdc9-191fa2e7b4df\") " pod="openshift-network-operator/iptables-alerter-855bl"
Jul 27 21:03:42.512325 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511647    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-os-release\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.512325 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511668    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72108aac-bf13-4304-8271-b32ad115d0ff-cni-binary-copy\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.512325 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511694    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/052f9040-58f7-4d2f-9909-bd9014cfd0af-socket-dir\") pod \"aws-ebs-csi-driver-node-85vpt\" (UID: \"052f9040-58f7-4d2f-9909-bd9014cfd0af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.512325 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511717    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/052f9040-58f7-4d2f-9909-bd9014cfd0af-etc-selinux\") pod \"aws-ebs-csi-driver-node-85vpt\" (UID: \"052f9040-58f7-4d2f-9909-bd9014cfd0af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.512325 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511743    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-var-lib-kubelet\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.512325 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511762    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-systemd-units\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.512325 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511789    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-etc-openvswitch\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.512325 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511819    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/72108aac-bf13-4304-8271-b32ad115d0ff-multus-daemon-config\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.512325 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511839    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-host-run-netns\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.512325 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511854    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d760a2e-48ae-4021-9766-7c875e7ec1c4-cnibin\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.512325 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511863    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/052f9040-58f7-4d2f-9909-bd9014cfd0af-socket-dir\") pod \"aws-ebs-csi-driver-node-85vpt\" (UID: \"052f9040-58f7-4d2f-9909-bd9014cfd0af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.512325 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511878    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d760a2e-48ae-4021-9766-7c875e7ec1c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.512325 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511898    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d760a2e-48ae-4021-9766-7c875e7ec1c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.512325 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511938    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/052f9040-58f7-4d2f-9909-bd9014cfd0af-sys-fs\") pod \"aws-ebs-csi-driver-node-85vpt\" (UID: \"052f9040-58f7-4d2f-9909-bd9014cfd0af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.512325 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511947    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-os-release\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.512325 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511956    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct6cp\" (UniqueName: \"kubernetes.io/projected/052f9040-58f7-4d2f-9909-bd9014cfd0af-kube-api-access-ct6cp\") pod \"aws-ebs-csi-driver-node-85vpt\" (UID: \"052f9040-58f7-4d2f-9909-bd9014cfd0af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.513164 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512000    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-run-openvswitch\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.513164 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.511900    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/052f9040-58f7-4d2f-9909-bd9014cfd0af-etc-selinux\") pod \"aws-ebs-csi-driver-node-85vpt\" (UID: \"052f9040-58f7-4d2f-9909-bd9014cfd0af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.513164 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512031    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d760a2e-48ae-4021-9766-7c875e7ec1c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.513164 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512076    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvt85\" (UniqueName: \"kubernetes.io/projected/f2f1d470-36dd-429a-bdc9-191fa2e7b4df-kube-api-access-jvt85\") pod \"iptables-alerter-855bl\" (UID: \"f2f1d470-36dd-429a-bdc9-191fa2e7b4df\") " pod="openshift-network-operator/iptables-alerter-855bl"
Jul 27 21:03:42.513164 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512113    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-host-run-k8s-cni-cncf-io\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.513164 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512114    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/052f9040-58f7-4d2f-9909-bd9014cfd0af-sys-fs\") pod \"aws-ebs-csi-driver-node-85vpt\" (UID: \"052f9040-58f7-4d2f-9909-bd9014cfd0af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.513164 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512181    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-host-run-k8s-cni-cncf-io\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.513164 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512202    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28lxq\" (UniqueName: \"kubernetes.io/projected/72108aac-bf13-4304-8271-b32ad115d0ff-kube-api-access-28lxq\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.513164 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512234    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-etc-sysconfig\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.513164 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512269    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-host-kubelet\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.513164 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512314    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-host-run-ovn-kubernetes\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.513164 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512335    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-ovnkube-script-lib\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.513164 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512360    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d760a2e-48ae-4021-9766-7c875e7ec1c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.513164 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512383    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/70daf650-e6e8-4170-83e4-a288b081605e-agent-certs\") pod \"konnectivity-agent-5fg9l\" (UID: \"70daf650-e6e8-4170-83e4-a288b081605e\") " pod="kube-system/konnectivity-agent-5fg9l"
Jul 27 21:03:42.513164 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512415    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-multus-conf-dir\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.513164 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512422    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/72108aac-bf13-4304-8271-b32ad115d0ff-multus-daemon-config\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.513164 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512444    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs\") pod \"network-metrics-daemon-whppm\" (UID: \"222ea6ea-61b8-4854-bc88-834449ce2974\") " pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:03:42.513875 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512424    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72108aac-bf13-4304-8271-b32ad115d0ff-cni-binary-copy\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.513875 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512467    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-etc-systemd\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.513875 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512474    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-multus-conf-dir\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.513875 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:42.512548    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:03:42.513875 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512581    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kn65\" (UniqueName: \"kubernetes.io/projected/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-kube-api-access-6kn65\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.513875 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512618    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-run\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.513875 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512643    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-sys\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.513875 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:42.512685    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs podName:222ea6ea-61b8-4854-bc88-834449ce2974 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:43.01265477 +0000 UTC m=+2.947871130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs") pod "network-metrics-daemon-whppm" (UID: "222ea6ea-61b8-4854-bc88-834449ce2974") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:03:42.513875 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512704    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-multus-cni-dir\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.513875 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512733    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-multus-socket-dir-parent\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.513875 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512759    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9d12f6c-3f55-44b3-b1d5-be1215730520-hosts-file\") pod \"node-resolver-gtpdk\" (UID: \"c9d12f6c-3f55-44b3-b1d5-be1215730520\") " pod="openshift-dns/node-resolver-gtpdk"
Jul 27 21:03:42.513875 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512796    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-etc-sysctl-conf\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.513875 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512821    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-run-systemd\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.513875 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512815    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-multus-socket-dir-parent\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.513875 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512860    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9d12f6c-3f55-44b3-b1d5-be1215730520-hosts-file\") pod \"node-resolver-gtpdk\" (UID: \"c9d12f6c-3f55-44b3-b1d5-be1215730520\") " pod="openshift-dns/node-resolver-gtpdk"
Jul 27 21:03:42.513875 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512867    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d760a2e-48ae-4021-9766-7c875e7ec1c4-system-cni-dir\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.513875 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512902    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret\") pod \"global-pull-secret-syncer-kvpv2\" (UID: \"71b80e11-ddee-4d86-a0b9-632732d0cca2\") " pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:03:42.514547 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512927    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-host-var-lib-cni-multus\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.514547 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512859    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-multus-cni-dir\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.514547 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512954    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-hostroot\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.514547 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.512993    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-host-var-lib-cni-multus\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.514547 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:42.513039    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:03:42.514547 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513087    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-hostroot\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.514547 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:42.513103    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret podName:71b80e11-ddee-4d86-a0b9-632732d0cca2 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:43.013087095 +0000 UTC m=+2.948303461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret") pod "global-pull-secret-syncer-kvpv2" (UID: "71b80e11-ddee-4d86-a0b9-632732d0cca2") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:03:42.514547 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513091    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-log-socket\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.514547 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513138    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/052f9040-58f7-4d2f-9909-bd9014cfd0af-registration-dir\") pod \"aws-ebs-csi-driver-node-85vpt\" (UID: \"052f9040-58f7-4d2f-9909-bd9014cfd0af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.514547 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513164    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-tmp\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.514547 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513191    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3de75e46-ec68-4801-9493-b1b31f67eefb-serviceca\") pod \"node-ca-cwvnf\" (UID: \"3de75e46-ec68-4801-9493-b1b31f67eefb\") " pod="openshift-image-registry/node-ca-cwvnf"
Jul 27 21:03:42.514547 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513219    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71b80e11-ddee-4d86-a0b9-632732d0cca2-kubelet-config\") pod \"global-pull-secret-syncer-kvpv2\" (UID: \"71b80e11-ddee-4d86-a0b9-632732d0cca2\") " pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:03:42.514547 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513254    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-var-lib-openvswitch\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.514547 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513259    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/052f9040-58f7-4d2f-9909-bd9014cfd0af-registration-dir\") pod \"aws-ebs-csi-driver-node-85vpt\" (UID: \"052f9040-58f7-4d2f-9909-bd9014cfd0af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.514547 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513316    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71b80e11-ddee-4d86-a0b9-632732d0cca2-kubelet-config\") pod \"global-pull-secret-syncer-kvpv2\" (UID: \"71b80e11-ddee-4d86-a0b9-632732d0cca2\") " pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:03:42.514547 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513347    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-node-log\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.514547 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513379    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f66sf\" (UniqueName: \"kubernetes.io/projected/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-kube-api-access-f66sf\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.515147 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513427    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-cnibin\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.515147 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513486    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-cnibin\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.515147 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513509    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-host-slash\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.515147 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513538    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-run-ovn\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.515147 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513578    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftr86\" (UniqueName: \"kubernetes.io/projected/222ea6ea-61b8-4854-bc88-834449ce2974-kube-api-access-ftr86\") pod \"network-metrics-daemon-whppm\" (UID: \"222ea6ea-61b8-4854-bc88-834449ce2974\") " pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:03:42.515147 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513607    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-etc-tuned\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.515147 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513691    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-host-cni-netd\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.515147 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513727    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/052f9040-58f7-4d2f-9909-bd9014cfd0af-device-dir\") pod \"aws-ebs-csi-driver-node-85vpt\" (UID: \"052f9040-58f7-4d2f-9909-bd9014cfd0af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.515147 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513784    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/052f9040-58f7-4d2f-9909-bd9014cfd0af-device-dir\") pod \"aws-ebs-csi-driver-node-85vpt\" (UID: \"052f9040-58f7-4d2f-9909-bd9014cfd0af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.515147 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513806    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-etc-modprobe-d\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.515147 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513833    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-etc-kubernetes\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.515147 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513859    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2f1d470-36dd-429a-bdc9-191fa2e7b4df-host-slash\") pod \"iptables-alerter-855bl\" (UID: \"f2f1d470-36dd-429a-bdc9-191fa2e7b4df\") " pod="openshift-network-operator/iptables-alerter-855bl"
Jul 27 21:03:42.515147 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513888    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71b80e11-ddee-4d86-a0b9-632732d0cca2-dbus\") pod \"global-pull-secret-syncer-kvpv2\" (UID: \"71b80e11-ddee-4d86-a0b9-632732d0cca2\") " pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:03:42.515147 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513912    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9d12f6c-3f55-44b3-b1d5-be1215730520-tmp-dir\") pod \"node-resolver-gtpdk\" (UID: \"c9d12f6c-3f55-44b3-b1d5-be1215730520\") " pod="openshift-dns/node-resolver-gtpdk"
Jul 27 21:03:42.515147 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513952    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-host\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.515147 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.513992    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-host-cni-bin\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.515147 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.514027    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71b80e11-ddee-4d86-a0b9-632732d0cca2-dbus\") pod \"global-pull-secret-syncer-kvpv2\" (UID: \"71b80e11-ddee-4d86-a0b9-632732d0cca2\") " pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:03:42.515710 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.514030    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.515710 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.514100    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-lib-modules\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.515710 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.514131    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-system-cni-dir\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.515710 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.514172    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-host-run-netns\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.515710 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.514182    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-system-cni-dir\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.515710 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.514216    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9d12f6c-3f55-44b3-b1d5-be1215730520-tmp-dir\") pod \"node-resolver-gtpdk\" (UID: \"c9d12f6c-3f55-44b3-b1d5-be1215730520\") " pod="openshift-dns/node-resolver-gtpdk"
Jul 27 21:03:42.515710 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.514268    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72108aac-bf13-4304-8271-b32ad115d0ff-host-run-netns\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.520069 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.519782    2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 21:03:42.520069 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:42.519796    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:03:42.520069 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:42.519816    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:03:42.520069 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:42.519827    2571 projected.go:196] Error preparing data for projected volume kube-api-access-v4rgh for pod openshift-network-diagnostics/network-check-target-m2bn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:03:42.520069 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:42.519865    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh podName:c268e820-791d-448a-b5ec-83ec69ad08ee nodeName:}" failed. No retries permitted until 2026-07-27 21:03:43.019852269 +0000 UTC m=+2.955068630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v4rgh" (UniqueName: "kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh") pod "network-check-target-m2bn8" (UID: "c268e820-791d-448a-b5ec-83ec69ad08ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:03:42.523828 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.523800    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct6cp\" (UniqueName: \"kubernetes.io/projected/052f9040-58f7-4d2f-9909-bd9014cfd0af-kube-api-access-ct6cp\") pod \"aws-ebs-csi-driver-node-85vpt\" (UID: \"052f9040-58f7-4d2f-9909-bd9014cfd0af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.524347 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.524327    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28lxq\" (UniqueName: \"kubernetes.io/projected/72108aac-bf13-4304-8271-b32ad115d0ff-kube-api-access-28lxq\") pod \"multus-x8x5c\" (UID: \"72108aac-bf13-4304-8271-b32ad115d0ff\") " pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.524445 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.524324    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkfgq\" (UniqueName: \"kubernetes.io/projected/c9d12f6c-3f55-44b3-b1d5-be1215730520-kube-api-access-pkfgq\") pod \"node-resolver-gtpdk\" (UID: \"c9d12f6c-3f55-44b3-b1d5-be1215730520\") " pod="openshift-dns/node-resolver-gtpdk"
Jul 27 21:03:42.524445 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.524437    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftr86\" (UniqueName: \"kubernetes.io/projected/222ea6ea-61b8-4854-bc88-834449ce2974-kube-api-access-ftr86\") pod \"network-metrics-daemon-whppm\" (UID: \"222ea6ea-61b8-4854-bc88-834449ce2974\") " pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:03:42.615276 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615177    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-var-lib-kubelet\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.615276 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615216    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-systemd-units\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.615276 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615232    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-etc-openvswitch\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.615276 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615249    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-host-run-netns\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.615276 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615274    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d760a2e-48ae-4021-9766-7c875e7ec1c4-cnibin\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.615671 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615301    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d760a2e-48ae-4021-9766-7c875e7ec1c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.615671 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615326    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d760a2e-48ae-4021-9766-7c875e7ec1c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.615671 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615339    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-systemd-units\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.615671 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615354    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-run-openvswitch\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.615671 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615369    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-var-lib-kubelet\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.615671 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615381    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d760a2e-48ae-4021-9766-7c875e7ec1c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.615671 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615339    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-host-run-netns\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.615671 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615411    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d760a2e-48ae-4021-9766-7c875e7ec1c4-cnibin\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.615671 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615412    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvt85\" (UniqueName: \"kubernetes.io/projected/f2f1d470-36dd-429a-bdc9-191fa2e7b4df-kube-api-access-jvt85\") pod \"iptables-alerter-855bl\" (UID: \"f2f1d470-36dd-429a-bdc9-191fa2e7b4df\") " pod="openshift-network-operator/iptables-alerter-855bl"
Jul 27 21:03:42.615671 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615451    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-etc-sysconfig\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.615671 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615477    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-host-kubelet\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.615671 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615506    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-host-run-ovn-kubernetes\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.615671 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615528    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-ovnkube-script-lib\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.615671 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615551    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d760a2e-48ae-4021-9766-7c875e7ec1c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.615671 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615576    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/70daf650-e6e8-4170-83e4-a288b081605e-agent-certs\") pod \"konnectivity-agent-5fg9l\" (UID: \"70daf650-e6e8-4170-83e4-a288b081605e\") " pod="kube-system/konnectivity-agent-5fg9l"
Jul 27 21:03:42.615671 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615613    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-etc-systemd\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.615671 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615641    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kn65\" (UniqueName: \"kubernetes.io/projected/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-kube-api-access-6kn65\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.616434 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615375    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-etc-openvswitch\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.616434 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615665    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-run\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.616434 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615690    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-sys\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.616434 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615715    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-etc-sysctl-conf\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.616434 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615741    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-run-systemd\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.616434 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615764    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d760a2e-48ae-4021-9766-7c875e7ec1c4-system-cni-dir\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.616434 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615811    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-log-socket\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.616434 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615837    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-tmp\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.616434 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615864    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3de75e46-ec68-4801-9493-b1b31f67eefb-serviceca\") pod \"node-ca-cwvnf\" (UID: \"3de75e46-ec68-4801-9493-b1b31f67eefb\") " pod="openshift-image-registry/node-ca-cwvnf"
Jul 27 21:03:42.616434 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615891    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-var-lib-openvswitch\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.616434 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615919    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-node-log\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.616434 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615946    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f66sf\" (UniqueName: \"kubernetes.io/projected/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-kube-api-access-f66sf\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.616434 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.615977    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-host-slash\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.616434 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616002    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-run-ovn\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.616434 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616059    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-etc-tuned\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.616434 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616085    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-host-cni-netd\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.616434 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616113    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-etc-modprobe-d\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.616434 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616108    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d760a2e-48ae-4021-9766-7c875e7ec1c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.617074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616143    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d760a2e-48ae-4021-9766-7c875e7ec1c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.617074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616164    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-run-openvswitch\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.617074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616204    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-etc-kubernetes\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.617074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616237    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2f1d470-36dd-429a-bdc9-191fa2e7b4df-host-slash\") pod \"iptables-alerter-855bl\" (UID: \"f2f1d470-36dd-429a-bdc9-191fa2e7b4df\") " pod="openshift-network-operator/iptables-alerter-855bl"
Jul 27 21:03:42.617074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616265    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-ovnkube-script-lib\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.617074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616267    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-host\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.617074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616309    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-host\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.617074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616313    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-host-cni-bin\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.617074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616351    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.617074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616357    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-etc-kubernetes\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.617074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616379    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-lib-modules\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.617074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616411    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2f1d470-36dd-429a-bdc9-191fa2e7b4df-host-slash\") pod \"iptables-alerter-855bl\" (UID: \"f2f1d470-36dd-429a-bdc9-191fa2e7b4df\") " pod="openshift-network-operator/iptables-alerter-855bl"
Jul 27 21:03:42.617074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616415    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-env-overrides\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.617074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616435    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-sys\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.617074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616445    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3de75e46-ec68-4801-9493-b1b31f67eefb-host\") pod \"node-ca-cwvnf\" (UID: \"3de75e46-ec68-4801-9493-b1b31f67eefb\") " pod="openshift-image-registry/node-ca-cwvnf"
Jul 27 21:03:42.617074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616461    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-run\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.617074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616481    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3de75e46-ec68-4801-9493-b1b31f67eefb-host\") pod \"node-ca-cwvnf\" (UID: \"3de75e46-ec68-4801-9493-b1b31f67eefb\") " pod="openshift-image-registry/node-ca-cwvnf"
Jul 27 21:03:42.617074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616484    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6gh\" (UniqueName: \"kubernetes.io/projected/3de75e46-ec68-4801-9493-b1b31f67eefb-kube-api-access-kr6gh\") pod \"node-ca-cwvnf\" (UID: \"3de75e46-ec68-4801-9493-b1b31f67eefb\") " pod="openshift-image-registry/node-ca-cwvnf"
Jul 27 21:03:42.618231 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616503    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-host-run-ovn-kubernetes\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.618231 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616519    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-host-cni-bin\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.618231 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616524    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-etc-sysctl-d\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.618231 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616557    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-ovnkube-config\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.618231 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616558    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.618231 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616579    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d760a2e-48ae-4021-9766-7c875e7ec1c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.618231 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616627    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-etc-systemd\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.618231 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616632    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d760a2e-48ae-4021-9766-7c875e7ec1c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.618231 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616682    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-lib-modules\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.618231 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616683    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-etc-sysconfig\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.618231 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616702    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-etc-sysctl-conf\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.618231 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616716    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-host-kubelet\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.618231 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616741    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-ovn-node-metrics-cert\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.618231 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616758    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-run-systemd\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.618231 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616770    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8d760a2e-48ae-4021-9766-7c875e7ec1c4-os-release\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.618231 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616803    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d760a2e-48ae-4021-9766-7c875e7ec1c4-system-cni-dir\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.618231 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616813    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcftf\" (UniqueName: \"kubernetes.io/projected/8d760a2e-48ae-4021-9766-7c875e7ec1c4-kube-api-access-vcftf\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.619117 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616845    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/70daf650-e6e8-4170-83e4-a288b081605e-konnectivity-ca\") pod \"konnectivity-agent-5fg9l\" (UID: \"70daf650-e6e8-4170-83e4-a288b081605e\") " pod="kube-system/konnectivity-agent-5fg9l"
Jul 27 21:03:42.619117 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616848    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-log-socket\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.619117 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616873    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f2f1d470-36dd-429a-bdc9-191fa2e7b4df-iptables-alerter-script\") pod \"iptables-alerter-855bl\" (UID: \"f2f1d470-36dd-429a-bdc9-191fa2e7b4df\") " pod="openshift-network-operator/iptables-alerter-855bl"
Jul 27 21:03:42.619117 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.616953    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-var-lib-openvswitch\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.619117 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.617108    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-env-overrides\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.619117 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.617318    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3de75e46-ec68-4801-9493-b1b31f67eefb-serviceca\") pod \"node-ca-cwvnf\" (UID: \"3de75e46-ec68-4801-9493-b1b31f67eefb\") " pod="openshift-image-registry/node-ca-cwvnf"
Jul 27 21:03:42.619117 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.617365    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-node-log\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.619117 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.617450    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-host-cni-netd\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.619117 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.617430    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f2f1d470-36dd-429a-bdc9-191fa2e7b4df-iptables-alerter-script\") pod \"iptables-alerter-855bl\" (UID: \"f2f1d470-36dd-429a-bdc9-191fa2e7b4df\") " pod="openshift-network-operator/iptables-alerter-855bl"
Jul 27 21:03:42.619117 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.617538    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-etc-modprobe-d\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.619117 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.617569    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-host-slash\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.619117 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.617688    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8d760a2e-48ae-4021-9766-7c875e7ec1c4-os-release\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.619117 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.617997    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-run-ovn\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.619117 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.618156    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-etc-sysctl-d\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.619117 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.618478    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/70daf650-e6e8-4170-83e4-a288b081605e-konnectivity-ca\") pod \"konnectivity-agent-5fg9l\" (UID: \"70daf650-e6e8-4170-83e4-a288b081605e\") " pod="kube-system/konnectivity-agent-5fg9l"
Jul 27 21:03:42.619117 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.618709    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-ovnkube-config\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.620182 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.620108    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-ovn-node-metrics-cert\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.620307 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.620283    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-tmp\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.620307 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.620300    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-etc-tuned\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.620455 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.620428    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/70daf650-e6e8-4170-83e4-a288b081605e-agent-certs\") pod \"konnectivity-agent-5fg9l\" (UID: \"70daf650-e6e8-4170-83e4-a288b081605e\") " pod="kube-system/konnectivity-agent-5fg9l"
Jul 27 21:03:42.624710 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.624685    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvt85\" (UniqueName: \"kubernetes.io/projected/f2f1d470-36dd-429a-bdc9-191fa2e7b4df-kube-api-access-jvt85\") pod \"iptables-alerter-855bl\" (UID: \"f2f1d470-36dd-429a-bdc9-191fa2e7b4df\") " pod="openshift-network-operator/iptables-alerter-855bl"
Jul 27 21:03:42.624902 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.624874    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kn65\" (UniqueName: \"kubernetes.io/projected/f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f-kube-api-access-6kn65\") pod \"tuned-fb28r\" (UID: \"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f\") " pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.625669 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.625646    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f66sf\" (UniqueName: \"kubernetes.io/projected/e2b3b600-ca21-4067-aa49-92df5b7b4bfa-kube-api-access-f66sf\") pod \"ovnkube-node-xkcbq\" (UID: \"e2b3b600-ca21-4067-aa49-92df5b7b4bfa\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.626777 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.626754    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcftf\" (UniqueName: \"kubernetes.io/projected/8d760a2e-48ae-4021-9766-7c875e7ec1c4-kube-api-access-vcftf\") pod \"multus-additional-cni-plugins-ttt2t\" (UID: \"8d760a2e-48ae-4021-9766-7c875e7ec1c4\") " pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.627171 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.627150    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr6gh\" (UniqueName: \"kubernetes.io/projected/3de75e46-ec68-4801-9493-b1b31f67eefb-kube-api-access-kr6gh\") pod \"node-ca-cwvnf\" (UID: \"3de75e46-ec68-4801-9493-b1b31f67eefb\") " pod="openshift-image-registry/node-ca-cwvnf"
Jul 27 21:03:42.706380 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.706344    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt"
Jul 27 21:03:42.716789 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.716764    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gtpdk"
Jul 27 21:03:42.723842 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.723813    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x8x5c"
Jul 27 21:03:42.728879 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.728852    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cwvnf"
Jul 27 21:03:42.734968 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.734944    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:03:42.743056 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.743035    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fb28r"
Jul 27 21:03:42.748945 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.748921    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ttt2t"
Jul 27 21:03:42.755805 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.755775    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5fg9l"
Jul 27 21:03:42.762337 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.761589    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-855bl"
Jul 27 21:03:42.784021 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:42.783997    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:03:43.020012 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:43.019984    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4rgh\" (UniqueName: \"kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh\") pod \"network-check-target-m2bn8\" (UID: \"c268e820-791d-448a-b5ec-83ec69ad08ee\") " pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:03:43.020183 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:43.020039    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs\") pod \"network-metrics-daemon-whppm\" (UID: \"222ea6ea-61b8-4854-bc88-834449ce2974\") " pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:03:43.020183 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:43.020077    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret\") pod \"global-pull-secret-syncer-kvpv2\" (UID: \"71b80e11-ddee-4d86-a0b9-632732d0cca2\") " pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:03:43.020300 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:43.020181    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:03:43.020300 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:43.020187    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:03:43.020300 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:43.020203    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:03:43.020300 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:43.020215    2571 projected.go:196] Error preparing data for projected volume kube-api-access-v4rgh for pod openshift-network-diagnostics/network-check-target-m2bn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:03:43.020300 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:43.020225    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:03:43.020300 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:43.020256    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret podName:71b80e11-ddee-4d86-a0b9-632732d0cca2 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:44.020235706 +0000 UTC m=+3.955452066 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret") pod "global-pull-secret-syncer-kvpv2" (UID: "71b80e11-ddee-4d86-a0b9-632732d0cca2") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:03:43.020300 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:43.020276    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh podName:c268e820-791d-448a-b5ec-83ec69ad08ee nodeName:}" failed. No retries permitted until 2026-07-27 21:03:44.020264943 +0000 UTC m=+3.955481302 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v4rgh" (UniqueName: "kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh") pod "network-check-target-m2bn8" (UID: "c268e820-791d-448a-b5ec-83ec69ad08ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:03:43.020300 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:43.020293    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs podName:222ea6ea-61b8-4854-bc88-834449ce2974 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:44.020284288 +0000 UTC m=+3.955500646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs") pod "network-metrics-daemon-whppm" (UID: "222ea6ea-61b8-4854-bc88-834449ce2974") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:03:43.143245 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:43.143060    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d760a2e_48ae_4021_9766_7c875e7ec1c4.slice/crio-b1f42bfad5078d9b72ea87ef5ffb2cee98234a02a1014040b8a7719a2b74d95a WatchSource:0}: Error finding container b1f42bfad5078d9b72ea87ef5ffb2cee98234a02a1014040b8a7719a2b74d95a: Status 404 returned error can't find the container with id b1f42bfad5078d9b72ea87ef5ffb2cee98234a02a1014040b8a7719a2b74d95a
Jul 27 21:03:43.144973 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:43.144705    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b3b600_ca21_4067_aa49_92df5b7b4bfa.slice/crio-e5b7b5e2fc60980beb69186100c9566abb56c7d753feee917692a5156923e21d WatchSource:0}: Error finding container e5b7b5e2fc60980beb69186100c9566abb56c7d753feee917692a5156923e21d: Status 404 returned error can't find the container with id e5b7b5e2fc60980beb69186100c9566abb56c7d753feee917692a5156923e21d
Jul 27 21:03:43.148315 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:43.148293    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70daf650_e6e8_4170_83e4_a288b081605e.slice/crio-0dcaa190c1e50510038fd51c0b0e9f782f764e1be7b36c3ea88fc322f97a69d9 WatchSource:0}: Error finding container 0dcaa190c1e50510038fd51c0b0e9f782f764e1be7b36c3ea88fc322f97a69d9: Status 404 returned error can't find the container with id 0dcaa190c1e50510038fd51c0b0e9f782f764e1be7b36c3ea88fc322f97a69d9
Jul 27 21:03:43.149017 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:43.148928    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de75e46_ec68_4801_9493_b1b31f67eefb.slice/crio-1b9276e18ed72dc0c57a044761468ad649600ccc8af25d7296e4d7f016ce6666 WatchSource:0}: Error finding container 1b9276e18ed72dc0c57a044761468ad649600ccc8af25d7296e4d7f016ce6666: Status 404 returned error can't find the container with id 1b9276e18ed72dc0c57a044761468ad649600ccc8af25d7296e4d7f016ce6666
Jul 27 21:03:43.149819 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:43.149665    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052f9040_58f7_4d2f_9909_bd9014cfd0af.slice/crio-df21bf30ae89d5d8d593d7a4b8b5c9f48baa3d965c08a6935fb8206b08ceb9ec WatchSource:0}: Error finding container df21bf30ae89d5d8d593d7a4b8b5c9f48baa3d965c08a6935fb8206b08ceb9ec: Status 404 returned error can't find the container with id df21bf30ae89d5d8d593d7a4b8b5c9f48baa3d965c08a6935fb8206b08ceb9ec
Jul 27 21:03:43.150700 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:43.150555    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72108aac_bf13_4304_8271_b32ad115d0ff.slice/crio-a431d6e535c54901675d573c83132d97af89757e755b3c189b4538c487a70f2a WatchSource:0}: Error finding container a431d6e535c54901675d573c83132d97af89757e755b3c189b4538c487a70f2a: Status 404 returned error can't find the container with id a431d6e535c54901675d573c83132d97af89757e755b3c189b4538c487a70f2a
Jul 27 21:03:43.151856 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:43.151834    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f1d470_36dd_429a_bdc9_191fa2e7b4df.slice/crio-2b0bb8cb3b9f6dde76eb84b569852de896eb5ad0b2e93531ea696246ab08f706 WatchSource:0}: Error finding container 2b0bb8cb3b9f6dde76eb84b569852de896eb5ad0b2e93531ea696246ab08f706: Status 404 returned error can't find the container with id 2b0bb8cb3b9f6dde76eb84b569852de896eb5ad0b2e93531ea696246ab08f706
Jul 27 21:03:43.155484 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:03:43.155463    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d12f6c_3f55_44b3_b1d5_be1215730520.slice/crio-c58188351e92abcfcafffbd9e4f8d205b4fe71a512d3f611f1d94a2c35a1736f WatchSource:0}: Error finding container c58188351e92abcfcafffbd9e4f8d205b4fe71a512d3f611f1d94a2c35a1736f: Status 404 returned error can't find the container with id c58188351e92abcfcafffbd9e4f8d205b4fe71a512d3f611f1d94a2c35a1736f
Jul 27 21:03:43.443909 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:43.443648    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:58:43 +0000 UTC" deadline="2028-05-13 14:51:00.060431966 +0000 UTC"
Jul 27 21:03:43.443909 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:43.443843    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15737h47m16.616592803s"
Jul 27 21:03:43.567910 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:43.567871    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:03:43.568113 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:43.568069    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-whppm" podUID="222ea6ea-61b8-4854-bc88-834449ce2974"
Jul 27 21:03:43.576036 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:43.576001    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x8x5c" event={"ID":"72108aac-bf13-4304-8271-b32ad115d0ff","Type":"ContainerStarted","Data":"a431d6e535c54901675d573c83132d97af89757e755b3c189b4538c487a70f2a"}
Jul 27 21:03:43.577812 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:43.577727    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cwvnf" event={"ID":"3de75e46-ec68-4801-9493-b1b31f67eefb","Type":"ContainerStarted","Data":"1b9276e18ed72dc0c57a044761468ad649600ccc8af25d7296e4d7f016ce6666"}
Jul 27 21:03:43.579228 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:43.579202    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5fg9l" event={"ID":"70daf650-e6e8-4170-83e4-a288b081605e","Type":"ContainerStarted","Data":"0dcaa190c1e50510038fd51c0b0e9f782f764e1be7b36c3ea88fc322f97a69d9"}
Jul 27 21:03:43.581130 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:43.581094    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq" event={"ID":"e2b3b600-ca21-4067-aa49-92df5b7b4bfa","Type":"ContainerStarted","Data":"e5b7b5e2fc60980beb69186100c9566abb56c7d753feee917692a5156923e21d"}
Jul 27 21:03:43.582950 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:43.582923    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fb28r" event={"ID":"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f","Type":"ContainerStarted","Data":"9a205e9b35cc6ed2fc7bea8954fb9ab9bbb475b8ea80c586b230065726a39cf1"}
Jul 27 21:03:43.585969 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:43.585922    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-50.ec2.internal" event={"ID":"fda18b08661970af357d9091388de747","Type":"ContainerStarted","Data":"1eda04c281b5ed702a038e0328697d4fba8dac9b1f7a48e7dacf69fc288b7818"}
Jul 27 21:03:43.588538 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:43.588511    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gtpdk" event={"ID":"c9d12f6c-3f55-44b3-b1d5-be1215730520","Type":"ContainerStarted","Data":"c58188351e92abcfcafffbd9e4f8d205b4fe71a512d3f611f1d94a2c35a1736f"}
Jul 27 21:03:43.591112 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:43.591083    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-855bl" event={"ID":"f2f1d470-36dd-429a-bdc9-191fa2e7b4df","Type":"ContainerStarted","Data":"2b0bb8cb3b9f6dde76eb84b569852de896eb5ad0b2e93531ea696246ab08f706"}
Jul 27 21:03:43.593682 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:43.593649    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttt2t" event={"ID":"8d760a2e-48ae-4021-9766-7c875e7ec1c4","Type":"ContainerStarted","Data":"b1f42bfad5078d9b72ea87ef5ffb2cee98234a02a1014040b8a7719a2b74d95a"}
Jul 27 21:03:43.596269 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:43.596241    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt" event={"ID":"052f9040-58f7-4d2f-9909-bd9014cfd0af","Type":"ContainerStarted","Data":"df21bf30ae89d5d8d593d7a4b8b5c9f48baa3d965c08a6935fb8206b08ceb9ec"}
Jul 27 21:03:43.602909 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:43.602854    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-50.ec2.internal" podStartSLOduration=1.602841442 podStartE2EDuration="1.602841442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:03:43.602258082 +0000 UTC m=+3.537474465" watchObservedRunningTime="2026-07-27 21:03:43.602841442 +0000 UTC m=+3.538057825"
Jul 27 21:03:44.028939 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:44.028892    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs\") pod \"network-metrics-daemon-whppm\" (UID: \"222ea6ea-61b8-4854-bc88-834449ce2974\") " pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:03:44.028939 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:44.028955    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret\") pod \"global-pull-secret-syncer-kvpv2\" (UID: \"71b80e11-ddee-4d86-a0b9-632732d0cca2\") " pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:03:44.029221 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:44.029031    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4rgh\" (UniqueName: \"kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh\") pod \"network-check-target-m2bn8\" (UID: \"c268e820-791d-448a-b5ec-83ec69ad08ee\") " pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:03:44.029221 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:44.029195    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:03:44.029221 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:44.029216    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:03:44.029401 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:44.029229    2571 projected.go:196] Error preparing data for projected volume kube-api-access-v4rgh for pod openshift-network-diagnostics/network-check-target-m2bn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:03:44.029401 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:44.029307    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh podName:c268e820-791d-448a-b5ec-83ec69ad08ee nodeName:}" failed. No retries permitted until 2026-07-27 21:03:46.029281671 +0000 UTC m=+5.964498044 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v4rgh" (UniqueName: "kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh") pod "network-check-target-m2bn8" (UID: "c268e820-791d-448a-b5ec-83ec69ad08ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:03:44.029401 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:44.029335    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:03:44.029401 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:44.029372    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:03:44.029637 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:44.029379    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs podName:222ea6ea-61b8-4854-bc88-834449ce2974 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:46.029366501 +0000 UTC m=+5.964582861 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs") pod "network-metrics-daemon-whppm" (UID: "222ea6ea-61b8-4854-bc88-834449ce2974") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:03:44.029637 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:44.029458    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret podName:71b80e11-ddee-4d86-a0b9-632732d0cca2 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:46.029439303 +0000 UTC m=+5.964655678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret") pod "global-pull-secret-syncer-kvpv2" (UID: "71b80e11-ddee-4d86-a0b9-632732d0cca2") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:03:44.444297 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:44.444203    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:58:43 +0000 UTC" deadline="2028-02-02 13:29:25.471548757 +0000 UTC"
Jul 27 21:03:44.444297 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:44.444247    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13312h25m41.02730691s"
Jul 27 21:03:44.569932 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:44.569897    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:03:44.572515 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:44.572492    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:03:44.572657 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:44.572595    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2bn8" podUID="c268e820-791d-448a-b5ec-83ec69ad08ee"
Jul 27 21:03:44.573440 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:44.573414    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kvpv2" podUID="71b80e11-ddee-4d86-a0b9-632732d0cca2"
Jul 27 21:03:45.567856 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:45.567827    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:03:45.568383 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:45.567957    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-whppm" podUID="222ea6ea-61b8-4854-bc88-834449ce2974"
Jul 27 21:03:46.047315 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:46.047274    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4rgh\" (UniqueName: \"kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh\") pod \"network-check-target-m2bn8\" (UID: \"c268e820-791d-448a-b5ec-83ec69ad08ee\") " pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:03:46.047516 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:46.047352    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs\") pod \"network-metrics-daemon-whppm\" (UID: \"222ea6ea-61b8-4854-bc88-834449ce2974\") " pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:03:46.047516 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:46.047390    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret\") pod \"global-pull-secret-syncer-kvpv2\" (UID: \"71b80e11-ddee-4d86-a0b9-632732d0cca2\") " pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:03:46.047625 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:46.047526    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:03:46.047625 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:46.047595    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret podName:71b80e11-ddee-4d86-a0b9-632732d0cca2 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:50.047575215 +0000 UTC m=+9.982791580 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret") pod "global-pull-secret-syncer-kvpv2" (UID: "71b80e11-ddee-4d86-a0b9-632732d0cca2") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:03:46.048056 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:46.048012    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:03:46.048056 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:46.048050    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:03:46.048201 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:46.048064    2571 projected.go:196] Error preparing data for projected volume kube-api-access-v4rgh for pod openshift-network-diagnostics/network-check-target-m2bn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:03:46.048201 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:46.048113    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh podName:c268e820-791d-448a-b5ec-83ec69ad08ee nodeName:}" failed. No retries permitted until 2026-07-27 21:03:50.048096214 +0000 UTC m=+9.983312578 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v4rgh" (UniqueName: "kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh") pod "network-check-target-m2bn8" (UID: "c268e820-791d-448a-b5ec-83ec69ad08ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:03:46.048201 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:46.048187    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:03:46.048357 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:46.048227    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs podName:222ea6ea-61b8-4854-bc88-834449ce2974 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:50.048213505 +0000 UTC m=+9.983429864 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs") pod "network-metrics-daemon-whppm" (UID: "222ea6ea-61b8-4854-bc88-834449ce2974") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:03:46.568106 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:46.568074    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:03:46.568607 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:46.568129    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:03:46.568607 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:46.568219    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2bn8" podUID="c268e820-791d-448a-b5ec-83ec69ad08ee"
Jul 27 21:03:46.568607 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:46.568314    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kvpv2" podUID="71b80e11-ddee-4d86-a0b9-632732d0cca2"
Jul 27 21:03:47.568175 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:47.568140    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:03:47.568702 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:47.568347    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-whppm" podUID="222ea6ea-61b8-4854-bc88-834449ce2974"
Jul 27 21:03:48.570354 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:48.569588    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:03:48.570354 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:48.569721    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kvpv2" podUID="71b80e11-ddee-4d86-a0b9-632732d0cca2"
Jul 27 21:03:48.570354 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:48.570176    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:03:48.570354 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:48.570273    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2bn8" podUID="c268e820-791d-448a-b5ec-83ec69ad08ee"
Jul 27 21:03:49.568303 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:49.568221    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:03:49.568473 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:49.568384    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-whppm" podUID="222ea6ea-61b8-4854-bc88-834449ce2974"
Jul 27 21:03:50.084760 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:50.084723    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret\") pod \"global-pull-secret-syncer-kvpv2\" (UID: \"71b80e11-ddee-4d86-a0b9-632732d0cca2\") " pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:03:50.085234 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:50.084796    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4rgh\" (UniqueName: \"kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh\") pod \"network-check-target-m2bn8\" (UID: \"c268e820-791d-448a-b5ec-83ec69ad08ee\") " pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:03:50.085234 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:50.084840    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs\") pod \"network-metrics-daemon-whppm\" (UID: \"222ea6ea-61b8-4854-bc88-834449ce2974\") " pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:03:50.085234 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:50.084949    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:03:50.085234 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:50.085041    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret podName:71b80e11-ddee-4d86-a0b9-632732d0cca2 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:58.085018897 +0000 UTC m=+18.020235260 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret") pod "global-pull-secret-syncer-kvpv2" (UID: "71b80e11-ddee-4d86-a0b9-632732d0cca2") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:03:50.085234 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:50.085092    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:03:50.085234 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:50.085113    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:03:50.085234 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:50.085126    2571 projected.go:196] Error preparing data for projected volume kube-api-access-v4rgh for pod openshift-network-diagnostics/network-check-target-m2bn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:03:50.085234 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:50.085151    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:03:50.085234 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:50.085238    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs podName:222ea6ea-61b8-4854-bc88-834449ce2974 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:58.085214987 +0000 UTC m=+18.020431360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs") pod "network-metrics-daemon-whppm" (UID: "222ea6ea-61b8-4854-bc88-834449ce2974") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:03:50.085656 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:50.085258    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh podName:c268e820-791d-448a-b5ec-83ec69ad08ee nodeName:}" failed. No retries permitted until 2026-07-27 21:03:58.085248081 +0000 UTC m=+18.020464454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v4rgh" (UniqueName: "kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh") pod "network-check-target-m2bn8" (UID: "c268e820-791d-448a-b5ec-83ec69ad08ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:03:50.569450 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:50.567158    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:03:50.569450 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:50.567287    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2bn8" podUID="c268e820-791d-448a-b5ec-83ec69ad08ee"
Jul 27 21:03:50.569450 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:50.567587    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:03:50.569450 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:50.567662    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kvpv2" podUID="71b80e11-ddee-4d86-a0b9-632732d0cca2"
Jul 27 21:03:51.567915 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:51.567878    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:03:51.568389 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:51.568026    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-whppm" podUID="222ea6ea-61b8-4854-bc88-834449ce2974"
Jul 27 21:03:52.567548 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:52.567517    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:03:52.567712 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:52.567626    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kvpv2" podUID="71b80e11-ddee-4d86-a0b9-632732d0cca2"
Jul 27 21:03:52.567712 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:52.567680    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:03:52.567810 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:52.567778    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2bn8" podUID="c268e820-791d-448a-b5ec-83ec69ad08ee"
Jul 27 21:03:53.567749 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:53.567712    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:03:53.568176 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:53.567842    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-whppm" podUID="222ea6ea-61b8-4854-bc88-834449ce2974"
Jul 27 21:03:54.570500 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:54.570466    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:03:54.570912 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:54.570596    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kvpv2" podUID="71b80e11-ddee-4d86-a0b9-632732d0cca2"
Jul 27 21:03:54.570912 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:54.570646    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:03:54.570912 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:54.570748    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2bn8" podUID="c268e820-791d-448a-b5ec-83ec69ad08ee"
Jul 27 21:03:55.567164 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:55.567124    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:03:55.567361 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:55.567271    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-whppm" podUID="222ea6ea-61b8-4854-bc88-834449ce2974"
Jul 27 21:03:56.568113 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:56.568070    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:03:56.568113 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:56.568106    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:03:56.568576 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:56.568238    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2bn8" podUID="c268e820-791d-448a-b5ec-83ec69ad08ee"
Jul 27 21:03:56.568576 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:56.568353    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kvpv2" podUID="71b80e11-ddee-4d86-a0b9-632732d0cca2"
Jul 27 21:03:57.567177 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:57.567137    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:03:57.567390 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:57.567267    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-whppm" podUID="222ea6ea-61b8-4854-bc88-834449ce2974"
Jul 27 21:03:58.145570 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:58.145533    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs\") pod \"network-metrics-daemon-whppm\" (UID: \"222ea6ea-61b8-4854-bc88-834449ce2974\") " pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:03:58.145976 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:58.145589    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret\") pod \"global-pull-secret-syncer-kvpv2\" (UID: \"71b80e11-ddee-4d86-a0b9-632732d0cca2\") " pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:03:58.145976 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:58.145640    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4rgh\" (UniqueName: \"kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh\") pod \"network-check-target-m2bn8\" (UID: \"c268e820-791d-448a-b5ec-83ec69ad08ee\") " pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:03:58.145976 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:58.145701    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:03:58.145976 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:58.145761    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:03:58.145976 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:58.145767    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:03:58.145976 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:58.145793    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:03:58.145976 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:58.145805    2571 projected.go:196] Error preparing data for projected volume kube-api-access-v4rgh for pod openshift-network-diagnostics/network-check-target-m2bn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:03:58.145976 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:58.145768    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs podName:222ea6ea-61b8-4854-bc88-834449ce2974 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:14.145751608 +0000 UTC m=+34.080967977 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs") pod "network-metrics-daemon-whppm" (UID: "222ea6ea-61b8-4854-bc88-834449ce2974") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:03:58.145976 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:58.145836    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret podName:71b80e11-ddee-4d86-a0b9-632732d0cca2 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:14.145817656 +0000 UTC m=+34.081034020 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret") pod "global-pull-secret-syncer-kvpv2" (UID: "71b80e11-ddee-4d86-a0b9-632732d0cca2") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:03:58.145976 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:58.145854    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh podName:c268e820-791d-448a-b5ec-83ec69ad08ee nodeName:}" failed. No retries permitted until 2026-07-27 21:04:14.145844808 +0000 UTC m=+34.081061170 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v4rgh" (UniqueName: "kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh") pod "network-check-target-m2bn8" (UID: "c268e820-791d-448a-b5ec-83ec69ad08ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:03:58.567815 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:58.567782    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:03:58.567815 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:58.567802    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:03:58.568030 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:58.567895    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2bn8" podUID="c268e820-791d-448a-b5ec-83ec69ad08ee"
Jul 27 21:03:58.568101 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:58.568056    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kvpv2" podUID="71b80e11-ddee-4d86-a0b9-632732d0cca2"
Jul 27 21:03:59.567800 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:03:59.567766    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:03:59.568230 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:03:59.567890    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-whppm" podUID="222ea6ea-61b8-4854-bc88-834449ce2974"
Jul 27 21:04:00.568153 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:00.568110    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:04:00.568619 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:00.568232    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2bn8" podUID="c268e820-791d-448a-b5ec-83ec69ad08ee"
Jul 27 21:04:00.568619 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:00.568331    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:04:00.568619 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:00.568428    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kvpv2" podUID="71b80e11-ddee-4d86-a0b9-632732d0cca2"
Jul 27 21:04:01.567466 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:01.567431    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:04:01.567651 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:01.567569    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-whppm" podUID="222ea6ea-61b8-4854-bc88-834449ce2974"
Jul 27 21:04:02.567731 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:02.567690    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:04:02.568217 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:02.567690    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:04:02.568217 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:02.567827    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kvpv2" podUID="71b80e11-ddee-4d86-a0b9-632732d0cca2"
Jul 27 21:04:02.568217 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:02.567902    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2bn8" podUID="c268e820-791d-448a-b5ec-83ec69ad08ee"
Jul 27 21:04:03.567660 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:03.567496    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:04:03.567793 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:03.567735    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-whppm" podUID="222ea6ea-61b8-4854-bc88-834449ce2974"
Jul 27 21:04:03.641308 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:03.641273    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5fg9l" event={"ID":"70daf650-e6e8-4170-83e4-a288b081605e","Type":"ContainerStarted","Data":"3f57fa8656100380035c0cae0e21ec4a1d11fd03f94569e23cbd83e8f466f7cc"}
Jul 27 21:04:03.643400 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:03.643367    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq" event={"ID":"e2b3b600-ca21-4067-aa49-92df5b7b4bfa","Type":"ContainerStarted","Data":"a181b2cd97e4dc32414f436fc8f78604d5d1f96c1dcd9fb47fd7fb9f515652d2"}
Jul 27 21:04:03.643531 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:03.643426    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq" event={"ID":"e2b3b600-ca21-4067-aa49-92df5b7b4bfa","Type":"ContainerStarted","Data":"885e5bd4351b9a4c876142ef65801d74afc0d24791804b7e5cd7fe7335e9c666"}
Jul 27 21:04:03.643531 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:03.643469    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq" event={"ID":"e2b3b600-ca21-4067-aa49-92df5b7b4bfa","Type":"ContainerStarted","Data":"f04ca3bbea859155936f9cb2c63aeba5cd278b3edcc02a0d886d5116f4c91be1"}
Jul 27 21:04:03.644684 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:03.644648    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fb28r" event={"ID":"f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f","Type":"ContainerStarted","Data":"0189d352988c0c670089ed9458f44b7377520b274bb7688b9ca87ae85b5f5d80"}
Jul 27 21:04:03.646050 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:03.646025    2571 generic.go:356] "Generic (PLEG): container finished" podID="ad061777f914723d1969e77779e70368" containerID="16dfacb6d4b7e09839925f50425c7cf607370d973f7695b1f0365c38ea5c580b" exitCode=0
Jul 27 21:04:03.646148 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:03.646114    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-50.ec2.internal" event={"ID":"ad061777f914723d1969e77779e70368","Type":"ContainerDied","Data":"16dfacb6d4b7e09839925f50425c7cf607370d973f7695b1f0365c38ea5c580b"}
Jul 27 21:04:03.647493 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:03.647475    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gtpdk" event={"ID":"c9d12f6c-3f55-44b3-b1d5-be1215730520","Type":"ContainerStarted","Data":"08788e026142622ca74a7cfd9825354406e5210b3ed8c138c4c6a0df064d0812"}
Jul 27 21:04:03.649070 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:03.649017    2571 generic.go:356] "Generic (PLEG): container finished" podID="8d760a2e-48ae-4021-9766-7c875e7ec1c4" containerID="9ea0d35f7c377926533e98d3a0aaddee4f4e1bffc3e4e2d1bc09d7476416ccea" exitCode=0
Jul 27 21:04:03.649158 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:03.649099    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttt2t" event={"ID":"8d760a2e-48ae-4021-9766-7c875e7ec1c4","Type":"ContainerDied","Data":"9ea0d35f7c377926533e98d3a0aaddee4f4e1bffc3e4e2d1bc09d7476416ccea"}
Jul 27 21:04:03.650497 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:03.650482    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt" event={"ID":"052f9040-58f7-4d2f-9909-bd9014cfd0af","Type":"ContainerStarted","Data":"9e595a30c7379292b4ea115f401ff6fbc03677d4c5ef253d2fb83dc75b7dd927"}
Jul 27 21:04:03.651817 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:03.651797    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x8x5c" event={"ID":"72108aac-bf13-4304-8271-b32ad115d0ff","Type":"ContainerStarted","Data":"033c62c136dea72b18549ec389926da171a84350dd8b86c1ddfcab4a5aadfdd4"}
Jul 27 21:04:03.654023 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:03.654005    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cwvnf" event={"ID":"3de75e46-ec68-4801-9493-b1b31f67eefb","Type":"ContainerStarted","Data":"50008fac3817027b701002c91bae4013fa8c692c15b6038da6f7a7e1e4b1c192"}
Jul 27 21:04:03.656892 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:03.656861    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5fg9l" podStartSLOduration=10.481581165 podStartE2EDuration="22.656851331s" totalImagesPullingTime="12.175270166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:03.656377755 +0000 UTC m=+23.591594139" watchObservedRunningTime="2026-07-27 21:04:03.656851331 +0000 UTC m=+23.592067709"
Jul 27 21:04:03.673591 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:03.673550    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fb28r" podStartSLOduration=2.663100534 podStartE2EDuration="22.673535605s" totalImagesPullingTime="20.010435071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:03.673372424 +0000 UTC m=+23.608588805" watchObservedRunningTime="2026-07-27 21:04:03.673535605 +0000 UTC m=+23.608751983"
Jul 27 21:04:03.712510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:03.712283    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-x8x5c" podStartSLOduration=2.657936958 podStartE2EDuration="22.712262687s" totalImagesPullingTime="20.054325729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:03.711223778 +0000 UTC m=+23.646440158" watchObservedRunningTime="2026-07-27 21:04:03.712262687 +0000 UTC m=+23.647479068"
Jul 27 21:04:03.726530 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:03.726481    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cwvnf" podStartSLOduration=2.7281082679999997 podStartE2EDuration="22.726463646s" totalImagesPullingTime="19.998355378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:03.726007897 +0000 UTC m=+23.661224275" watchObservedRunningTime="2026-07-27 21:04:03.726463646 +0000 UTC m=+23.661680025"
Jul 27 21:04:03.754430 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:03.754391    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gtpdk" podStartSLOduration=2.759408298 podStartE2EDuration="22.754372067s" totalImagesPullingTime="19.994963769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:03.754097094 +0000 UTC m=+23.689313472" watchObservedRunningTime="2026-07-27 21:04:03.754372067 +0000 UTC m=+23.689588449"
Jul 27 21:04:03.939144 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:03.939111    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5fg9l"
Jul 27 21:04:03.939688 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:03.939668    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5fg9l"
Jul 27 21:04:04.433874 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:04.433793    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:04.520446 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:04.520418    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:04.571345 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:04.571319    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:04:04.571345 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:04.571332    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:04:04.571840 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:04.571417    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kvpv2" podUID="71b80e11-ddee-4d86-a0b9-632732d0cca2"
Jul 27 21:04:04.571840 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:04.571560    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2bn8" podUID="c268e820-791d-448a-b5ec-83ec69ad08ee"
Jul 27 21:04:04.658376 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:04.658342    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-50.ec2.internal" event={"ID":"ad061777f914723d1969e77779e70368","Type":"ContainerStarted","Data":"5acf412d6b55498dbab136b35bac90703651755907b667d43cddb580da7c88ff"}
Jul 27 21:04:04.660192 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:04.660143    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-855bl" event={"ID":"f2f1d470-36dd-429a-bdc9-191fa2e7b4df","Type":"ContainerStarted","Data":"95184d6b22060ada0c4f95ed101aa324b2a2eaa2b6a3c9d53443bec8f2e6f706"}
Jul 27 21:04:04.663431 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:04.663410    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq" event={"ID":"e2b3b600-ca21-4067-aa49-92df5b7b4bfa","Type":"ContainerStarted","Data":"05f101f3e327326b122c0d17f149ff884974232bde93df2608040a5650abe282"}
Jul 27 21:04:04.663552 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:04.663482    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq" event={"ID":"e2b3b600-ca21-4067-aa49-92df5b7b4bfa","Type":"ContainerStarted","Data":"6f9c1df22f9c66bccf2cdc19e536e9ee362f75180786c1514cb23b294b86e18b"}
Jul 27 21:04:04.663552 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:04.663534    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq" event={"ID":"e2b3b600-ca21-4067-aa49-92df5b7b4bfa","Type":"ContainerStarted","Data":"8a8a592808e1e3377d97419a7d94a2f23e307b996a3e84a8e495be1195c3071d"}
Jul 27 21:04:04.664181 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:04.664163    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5fg9l"
Jul 27 21:04:04.664662 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:04.664642    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5fg9l"
Jul 27 21:04:04.675696 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:04.675652    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-50.ec2.internal" podStartSLOduration=22.675637086000002 podStartE2EDuration="22.675637086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:04.675219213 +0000 UTC m=+24.610435592" watchObservedRunningTime="2026-07-27 21:04:04.675637086 +0000 UTC m=+24.610853466"
Jul 27 21:04:04.689063 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:04.688978    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-855bl" podStartSLOduration=3.6924908629999997 podStartE2EDuration="23.688940596s" totalImagesPullingTime="19.996449733s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:04.688483351 +0000 UTC m=+24.623699732" watchObservedRunningTime="2026-07-27 21:04:04.688940596 +0000 UTC m=+24.624156975"
Jul 27 21:04:04.763881 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:04.763826    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:04.824783 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:04.824751    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:04.858640 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:04.858611    2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 21:04:04.929268 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:04.929234    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:05.466311 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:05.466204    2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T21:04:04.858636315Z","UUID":"1cdfb0e3-0e4c-4ec7-911b-48b46083f62c","Handler":null,"Name":"","Endpoint":""}
Jul 27 21:04:05.468067 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:05.468041    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 21:04:05.468206 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:05.468083    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 21:04:05.567770 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:05.567729    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:04:05.567923 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:05.567865    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-whppm" podUID="222ea6ea-61b8-4854-bc88-834449ce2974"
Jul 27 21:04:05.668481 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:05.668446    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt" event={"ID":"052f9040-58f7-4d2f-9909-bd9014cfd0af","Type":"ContainerStarted","Data":"ae398660f1dcac743c341cdf567d9aad81f5639aafedcb82daa0f224513759e8"}
Jul 27 21:04:06.567639 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:06.567600    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:04:06.567840 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:06.567603    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:04:06.567840 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:06.567750    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2bn8" podUID="c268e820-791d-448a-b5ec-83ec69ad08ee"
Jul 27 21:04:06.567946 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:06.567833    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kvpv2" podUID="71b80e11-ddee-4d86-a0b9-632732d0cca2"
Jul 27 21:04:06.672419 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:06.672381    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt" event={"ID":"052f9040-58f7-4d2f-9909-bd9014cfd0af","Type":"ContainerStarted","Data":"f8e269c15ef375ffb9f9edbbf110cc561bbc695384f7db43506fcdbad7e186bd"}
Jul 27 21:04:06.675890 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:06.675851    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq" event={"ID":"e2b3b600-ca21-4067-aa49-92df5b7b4bfa","Type":"ContainerStarted","Data":"bee21345c9ce23e0fea6bf58ac930fa2b31a9d6f19baddec915c271d939c6445"}
Jul 27 21:04:06.691903 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:06.691843    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85vpt" podStartSLOduration=3.50546483 podStartE2EDuration="25.691827199s" totalImagesPullingTime="22.186362369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:06.691548654 +0000 UTC m=+26.626765033" watchObservedRunningTime="2026-07-27 21:04:06.691827199 +0000 UTC m=+26.627043578"
Jul 27 21:04:07.536506 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:07.536475    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:07.567482 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:07.567453    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:04:07.567674 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:07.567573    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-whppm" podUID="222ea6ea-61b8-4854-bc88-834449ce2974"
Jul 27 21:04:07.638616 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:07.638580    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:07.741005 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:07.740949    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:07.839871 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:07.839785    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:08.570657 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:08.570500    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:04:08.570804 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:08.570563    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:04:08.570804 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:08.570726    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kvpv2" podUID="71b80e11-ddee-4d86-a0b9-632732d0cca2"
Jul 27 21:04:08.570891 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:08.570849    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2bn8" podUID="c268e820-791d-448a-b5ec-83ec69ad08ee"
Jul 27 21:04:08.684279 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:08.684241    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq" event={"ID":"e2b3b600-ca21-4067-aa49-92df5b7b4bfa","Type":"ContainerStarted","Data":"e4fe5c4828e7bc149e12898f6c14921d33536412b4f7465aed6c23020cbb48b1"}
Jul 27 21:04:08.684499 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:08.684413    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:04:08.684499 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:08.684471    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:04:08.686118 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:08.686092    2571 generic.go:356] "Generic (PLEG): container finished" podID="8d760a2e-48ae-4021-9766-7c875e7ec1c4" containerID="443356e1d7a8936452308645c7a7314254f3a61a38c41ae0b8b1897133171696" exitCode=0
Jul 27 21:04:08.686233 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:08.686138    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttt2t" event={"ID":"8d760a2e-48ae-4021-9766-7c875e7ec1c4","Type":"ContainerDied","Data":"443356e1d7a8936452308645c7a7314254f3a61a38c41ae0b8b1897133171696"}
Jul 27 21:04:08.701030 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:08.701006    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:04:08.714454 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:08.712419    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq" podStartSLOduration=7.648760156 podStartE2EDuration="27.712400648s" totalImagesPullingTime="20.063640492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:08.711574357 +0000 UTC m=+28.646790736" watchObservedRunningTime="2026-07-27 21:04:08.712400648 +0000 UTC m=+28.647617027"
Jul 27 21:04:09.567871 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:09.567836    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:04:09.568604 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:09.567948    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-whppm" podUID="222ea6ea-61b8-4854-bc88-834449ce2974"
Jul 27 21:04:09.689093 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:09.689062    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:04:09.704100 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:09.704074    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:04:10.204109 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:10.203824    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-whppm"]
Jul 27 21:04:10.204260 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:10.204177    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:04:10.204329 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:10.204306    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-whppm" podUID="222ea6ea-61b8-4854-bc88-834449ce2974"
Jul 27 21:04:10.206877 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:10.206847    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m2bn8"]
Jul 27 21:04:10.207044 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:10.207013    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:04:10.207136 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:10.207114    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2bn8" podUID="c268e820-791d-448a-b5ec-83ec69ad08ee"
Jul 27 21:04:10.207495 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:10.207475    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kvpv2"]
Jul 27 21:04:10.207589 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:10.207553    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:04:10.207650 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:10.207633    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kvpv2" podUID="71b80e11-ddee-4d86-a0b9-632732d0cca2"
Jul 27 21:04:10.693209 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:10.693177    2571 generic.go:356] "Generic (PLEG): container finished" podID="8d760a2e-48ae-4021-9766-7c875e7ec1c4" containerID="7af842331850a21602676008335cc876a57bf9ee5b74443b2c6708abcaabc354" exitCode=0
Jul 27 21:04:10.693590 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:10.693263    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttt2t" event={"ID":"8d760a2e-48ae-4021-9766-7c875e7ec1c4","Type":"ContainerDied","Data":"7af842331850a21602676008335cc876a57bf9ee5b74443b2c6708abcaabc354"}
Jul 27 21:04:10.865130 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:10.865101    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:10.965057 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:10.964977    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:11.567770 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:11.567740    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:04:11.567919 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:11.567840    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2bn8" podUID="c268e820-791d-448a-b5ec-83ec69ad08ee"
Jul 27 21:04:11.567919 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:11.567746    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:04:11.567919 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:11.567758    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:04:11.568038 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:11.567929    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kvpv2" podUID="71b80e11-ddee-4d86-a0b9-632732d0cca2"
Jul 27 21:04:11.568073 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:11.568031    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-whppm" podUID="222ea6ea-61b8-4854-bc88-834449ce2974"
Jul 27 21:04:12.699971 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:12.699923    2571 generic.go:356] "Generic (PLEG): container finished" podID="8d760a2e-48ae-4021-9766-7c875e7ec1c4" containerID="1c5749aca4835d2aadd343ed7731a06da25fbca514f5816619c3e29cd1429974" exitCode=0
Jul 27 21:04:12.700314 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:12.699989    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttt2t" event={"ID":"8d760a2e-48ae-4021-9766-7c875e7ec1c4","Type":"ContainerDied","Data":"1c5749aca4835d2aadd343ed7731a06da25fbca514f5816619c3e29cd1429974"}
Jul 27 21:04:13.434823 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:13.434788    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:13.528365 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:13.528307    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:13.567573 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:13.567535    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:04:13.567736 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:13.567633    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:04:13.567736 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:13.567649    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2bn8" podUID="c268e820-791d-448a-b5ec-83ec69ad08ee"
Jul 27 21:04:13.567736 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:13.567670    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:04:13.567853 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:13.567740    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kvpv2" podUID="71b80e11-ddee-4d86-a0b9-632732d0cca2"
Jul 27 21:04:13.567853 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:13.567816    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-whppm" podUID="222ea6ea-61b8-4854-bc88-834449ce2974"
Jul 27 21:04:13.610204 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:13.610164    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:13.695208 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:13.695114    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:13.796557 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:13.796501    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:13.874501 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:13.874466    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:14.167886 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:14.167856    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs\") pod \"network-metrics-daemon-whppm\" (UID: \"222ea6ea-61b8-4854-bc88-834449ce2974\") " pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:04:14.167886 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:14.167893    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret\") pod \"global-pull-secret-syncer-kvpv2\" (UID: \"71b80e11-ddee-4d86-a0b9-632732d0cca2\") " pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:04:14.168153 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:14.167935    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4rgh\" (UniqueName: \"kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh\") pod \"network-check-target-m2bn8\" (UID: \"c268e820-791d-448a-b5ec-83ec69ad08ee\") " pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:04:14.168153 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:14.168031    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:14.168153 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:14.168064    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:14.168153 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:14.168071    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:14.168153 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:14.168098    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:14.168153 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:14.168111    2571 projected.go:196] Error preparing data for projected volume kube-api-access-v4rgh for pod openshift-network-diagnostics/network-check-target-m2bn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:14.168153 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:14.168110    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs podName:222ea6ea-61b8-4854-bc88-834449ce2974 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:46.168090174 +0000 UTC m=+66.103306533 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs") pod "network-metrics-daemon-whppm" (UID: "222ea6ea-61b8-4854-bc88-834449ce2974") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:14.168428 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:14.168167    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret podName:71b80e11-ddee-4d86-a0b9-632732d0cca2 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:46.168151405 +0000 UTC m=+66.103367765 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret") pod "global-pull-secret-syncer-kvpv2" (UID: "71b80e11-ddee-4d86-a0b9-632732d0cca2") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:14.168428 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:14.168182    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh podName:c268e820-791d-448a-b5ec-83ec69ad08ee nodeName:}" failed. No retries permitted until 2026-07-27 21:04:46.168173163 +0000 UTC m=+66.103389521 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-v4rgh" (UniqueName: "kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh") pod "network-check-target-m2bn8" (UID: "c268e820-791d-448a-b5ec-83ec69ad08ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:15.567542 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:15.567497    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:04:15.567542 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:15.567529    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:04:15.568100 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:15.567636    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kvpv2" podUID="71b80e11-ddee-4d86-a0b9-632732d0cca2"
Jul 27 21:04:15.568100 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:15.567653    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:04:15.568100 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:15.567743    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-whppm" podUID="222ea6ea-61b8-4854-bc88-834449ce2974"
Jul 27 21:04:15.568100 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:15.567840    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2bn8" podUID="c268e820-791d-448a-b5ec-83ec69ad08ee"
Jul 27 21:04:15.920496 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:15.920268    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-50.ec2.internal" event="NodeReady"
Jul 27 21:04:15.920648 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:15.920582    2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 21:04:15.999853 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:15.998920    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58ff8dd5bb-fv56s"]
Jul 27 21:04:16.001694 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.001664    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rc6n5"]
Jul 27 21:04:16.001798 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.001774    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.003829 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.003809    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xpx7g"]
Jul 27 21:04:16.004055 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.004035    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rc6n5"
Jul 27 21:04:16.004276 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.004258    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 27 21:04:16.004629 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.004613    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 27 21:04:16.004701 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.004618    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l946f\""
Jul 27 21:04:16.004701 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.004671    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 27 21:04:16.006314 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.006280    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 21:04:16.009753 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.007231    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 21:04:16.009753 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.007610    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jkr4s\""
Jul 27 21:04:16.009753 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.007690    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xpx7g"
Jul 27 21:04:16.010882 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.010689    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 21:04:16.010882 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.010740    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 21:04:16.010882 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.010689    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xwcxr\""
Jul 27 21:04:16.010882 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.010744    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 21:04:16.012502 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.012269    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 27 21:04:16.017765 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.017746    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rc6n5"]
Jul 27 21:04:16.018117 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.018098    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xpx7g"]
Jul 27 21:04:16.020085 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.018922    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58ff8dd5bb-fv56s"]
Jul 27 21:04:16.131820 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.131783    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:16.184583 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.184491    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40a3772d-f99d-45ae-bb4e-3bd833d47da0-tmp-dir\") pod \"dns-default-rc6n5\" (UID: \"40a3772d-f99d-45ae-bb4e-3bd833d47da0\") " pod="openshift-dns/dns-default-rc6n5"
Jul 27 21:04:16.184583 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.184537    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-ca-trust-extracted\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.184583 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.184572    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-image-registry-private-configuration\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.184812 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.184602    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-tls\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.184812 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.184619    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-installation-pull-secrets\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.184812 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.184634    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-bound-sa-token\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.184812 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.184660    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40a3772d-f99d-45ae-bb4e-3bd833d47da0-metrics-tls\") pod \"dns-default-rc6n5\" (UID: \"40a3772d-f99d-45ae-bb4e-3bd833d47da0\") " pod="openshift-dns/dns-default-rc6n5"
Jul 27 21:04:16.184812 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.184746    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-certificates\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.184812 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.184786    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rfvn\" (UniqueName: \"kubernetes.io/projected/5d872ddb-8292-4ecc-b10b-98e6c884724e-kube-api-access-5rfvn\") pod \"ingress-canary-xpx7g\" (UID: \"5d872ddb-8292-4ecc-b10b-98e6c884724e\") " pod="openshift-ingress-canary/ingress-canary-xpx7g"
Jul 27 21:04:16.185095 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.184813    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/40a3772d-f99d-45ae-bb4e-3bd833d47da0-config-volume\") pod \"dns-default-rc6n5\" (UID: \"40a3772d-f99d-45ae-bb4e-3bd833d47da0\") " pod="openshift-dns/dns-default-rc6n5"
Jul 27 21:04:16.185095 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.184850    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-trusted-ca\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.185095 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.184891    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thx25\" (UniqueName: \"kubernetes.io/projected/40a3772d-f99d-45ae-bb4e-3bd833d47da0-kube-api-access-thx25\") pod \"dns-default-rc6n5\" (UID: \"40a3772d-f99d-45ae-bb4e-3bd833d47da0\") " pod="openshift-dns/dns-default-rc6n5"
Jul 27 21:04:16.185095 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.184941    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d872ddb-8292-4ecc-b10b-98e6c884724e-cert\") pod \"ingress-canary-xpx7g\" (UID: \"5d872ddb-8292-4ecc-b10b-98e6c884724e\") " pod="openshift-ingress-canary/ingress-canary-xpx7g"
Jul 27 21:04:16.185095 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.184984    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgcfg\" (UniqueName: \"kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-kube-api-access-vgcfg\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.232101 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.232070    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:16.285555 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.285516    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rfvn\" (UniqueName: \"kubernetes.io/projected/5d872ddb-8292-4ecc-b10b-98e6c884724e-kube-api-access-5rfvn\") pod \"ingress-canary-xpx7g\" (UID: \"5d872ddb-8292-4ecc-b10b-98e6c884724e\") " pod="openshift-ingress-canary/ingress-canary-xpx7g"
Jul 27 21:04:16.285743 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.285563    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/40a3772d-f99d-45ae-bb4e-3bd833d47da0-config-volume\") pod \"dns-default-rc6n5\" (UID: \"40a3772d-f99d-45ae-bb4e-3bd833d47da0\") " pod="openshift-dns/dns-default-rc6n5"
Jul 27 21:04:16.285743 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.285591    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-trusted-ca\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.285743 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.285618    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thx25\" (UniqueName: \"kubernetes.io/projected/40a3772d-f99d-45ae-bb4e-3bd833d47da0-kube-api-access-thx25\") pod \"dns-default-rc6n5\" (UID: \"40a3772d-f99d-45ae-bb4e-3bd833d47da0\") " pod="openshift-dns/dns-default-rc6n5"
Jul 27 21:04:16.285743 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.285653    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d872ddb-8292-4ecc-b10b-98e6c884724e-cert\") pod \"ingress-canary-xpx7g\" (UID: \"5d872ddb-8292-4ecc-b10b-98e6c884724e\") " pod="openshift-ingress-canary/ingress-canary-xpx7g"
Jul 27 21:04:16.285743 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.285682    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgcfg\" (UniqueName: \"kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-kube-api-access-vgcfg\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.285743 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.285730    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40a3772d-f99d-45ae-bb4e-3bd833d47da0-tmp-dir\") pod \"dns-default-rc6n5\" (UID: \"40a3772d-f99d-45ae-bb4e-3bd833d47da0\") " pod="openshift-dns/dns-default-rc6n5"
Jul 27 21:04:16.286081 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.285754    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-ca-trust-extracted\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.286081 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.285786    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-image-registry-private-configuration\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.286081 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.285815    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-tls\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.286081 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.285839    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-installation-pull-secrets\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.286081 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.285863    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-bound-sa-token\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.286081 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.285889    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40a3772d-f99d-45ae-bb4e-3bd833d47da0-metrics-tls\") pod \"dns-default-rc6n5\" (UID: \"40a3772d-f99d-45ae-bb4e-3bd833d47da0\") " pod="openshift-dns/dns-default-rc6n5"
Jul 27 21:04:16.286081 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.285932    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-certificates\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.286410 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.286132    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40a3772d-f99d-45ae-bb4e-3bd833d47da0-tmp-dir\") pod \"dns-default-rc6n5\" (UID: \"40a3772d-f99d-45ae-bb4e-3bd833d47da0\") " pod="openshift-dns/dns-default-rc6n5"
Jul 27 21:04:16.286410 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.286296    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/40a3772d-f99d-45ae-bb4e-3bd833d47da0-config-volume\") pod \"dns-default-rc6n5\" (UID: \"40a3772d-f99d-45ae-bb4e-3bd833d47da0\") " pod="openshift-dns/dns-default-rc6n5"
Jul 27 21:04:16.286410 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:16.286405    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:04:16.286550 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:16.286420    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58ff8dd5bb-fv56s: secret "image-registry-tls" not found
Jul 27 21:04:16.286550 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.286427    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-certificates\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.286550 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:16.286487    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-tls podName:f6cf37cf-d28f-47fe-8a27-6585ab6e1579 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:16.786469063 +0000 UTC m=+36.721685451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-tls") pod "image-registry-58ff8dd5bb-fv56s" (UID: "f6cf37cf-d28f-47fe-8a27-6585ab6e1579") : secret "image-registry-tls" not found
Jul 27 21:04:16.286783 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.286762    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-trusted-ca\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.286990 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.286950    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-ca-trust-extracted\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.287068 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:16.287003    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:04:16.287068 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:16.287067    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40a3772d-f99d-45ae-bb4e-3bd833d47da0-metrics-tls podName:40a3772d-f99d-45ae-bb4e-3bd833d47da0 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:16.787050046 +0000 UTC m=+36.722266409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/40a3772d-f99d-45ae-bb4e-3bd833d47da0-metrics-tls") pod "dns-default-rc6n5" (UID: "40a3772d-f99d-45ae-bb4e-3bd833d47da0") : secret "dns-default-metrics-tls" not found
Jul 27 21:04:16.291351 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.291329    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-image-registry-private-configuration\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.291351 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.291341    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-installation-pull-secrets\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.292086 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.292069    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d872ddb-8292-4ecc-b10b-98e6c884724e-cert\") pod \"ingress-canary-xpx7g\" (UID: \"5d872ddb-8292-4ecc-b10b-98e6c884724e\") " pod="openshift-ingress-canary/ingress-canary-xpx7g"
Jul 27 21:04:16.295434 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.295411    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thx25\" (UniqueName: \"kubernetes.io/projected/40a3772d-f99d-45ae-bb4e-3bd833d47da0-kube-api-access-thx25\") pod \"dns-default-rc6n5\" (UID: \"40a3772d-f99d-45ae-bb4e-3bd833d47da0\") " pod="openshift-dns/dns-default-rc6n5"
Jul 27 21:04:16.295503 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.295467    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rfvn\" (UniqueName: \"kubernetes.io/projected/5d872ddb-8292-4ecc-b10b-98e6c884724e-kube-api-access-5rfvn\") pod \"ingress-canary-xpx7g\" (UID: \"5d872ddb-8292-4ecc-b10b-98e6c884724e\") " pod="openshift-ingress-canary/ingress-canary-xpx7g"
Jul 27 21:04:16.295619 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.295583    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-bound-sa-token\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.295932 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.295909    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgcfg\" (UniqueName: \"kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-kube-api-access-vgcfg\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.329458 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.329421    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xpx7g"
Jul 27 21:04:16.686002 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.685946    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xpx7g"]
Jul 27 21:04:16.692650 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:04:16.692623    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d872ddb_8292_4ecc_b10b_98e6c884724e.slice/crio-3c83befaf21dc7eb4895fd7afe9de13a0b52e68f60d2734865e4dc4e6475990c WatchSource:0}: Error finding container 3c83befaf21dc7eb4895fd7afe9de13a0b52e68f60d2734865e4dc4e6475990c: Status 404 returned error can't find the container with id 3c83befaf21dc7eb4895fd7afe9de13a0b52e68f60d2734865e4dc4e6475990c
Jul 27 21:04:16.710186 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.710153    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xpx7g" event={"ID":"5d872ddb-8292-4ecc-b10b-98e6c884724e","Type":"ContainerStarted","Data":"3c83befaf21dc7eb4895fd7afe9de13a0b52e68f60d2734865e4dc4e6475990c"}
Jul 27 21:04:16.713432 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.713400    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttt2t" event={"ID":"8d760a2e-48ae-4021-9766-7c875e7ec1c4","Type":"ContainerStarted","Data":"b2473a61cb7dc2c0b7448401ec719f5b5f6ba07c4b7ae8935bdb4b25cb6ed1c7"}
Jul 27 21:04:16.789886 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.789860    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-tls\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:16.789980 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:16.789900    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40a3772d-f99d-45ae-bb4e-3bd833d47da0-metrics-tls\") pod \"dns-default-rc6n5\" (UID: \"40a3772d-f99d-45ae-bb4e-3bd833d47da0\") " pod="openshift-dns/dns-default-rc6n5"
Jul 27 21:04:16.790063 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:16.790044    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:04:16.790103 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:16.790066    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58ff8dd5bb-fv56s: secret "image-registry-tls" not found
Jul 27 21:04:16.790103 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:16.790065    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:04:16.790170 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:16.790124    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-tls podName:f6cf37cf-d28f-47fe-8a27-6585ab6e1579 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:17.790109038 +0000 UTC m=+37.725325394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-tls") pod "image-registry-58ff8dd5bb-fv56s" (UID: "f6cf37cf-d28f-47fe-8a27-6585ab6e1579") : secret "image-registry-tls" not found
Jul 27 21:04:16.790170 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:16.790138    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40a3772d-f99d-45ae-bb4e-3bd833d47da0-metrics-tls podName:40a3772d-f99d-45ae-bb4e-3bd833d47da0 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:17.790132212 +0000 UTC m=+37.725348570 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/40a3772d-f99d-45ae-bb4e-3bd833d47da0-metrics-tls") pod "dns-default-rc6n5" (UID: "40a3772d-f99d-45ae-bb4e-3bd833d47da0") : secret "dns-default-metrics-tls" not found
Jul 27 21:04:17.568033 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:17.567758    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:04:17.568033 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:17.567761    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:04:17.568200 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:17.567792    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:04:17.570824 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:17.570804    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 21:04:17.571741 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:17.571722    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 21:04:17.571741 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:17.571735    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bztw5\""
Jul 27 21:04:17.571918 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:17.571762    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 21:04:17.571918 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:17.571738    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ww7jp\""
Jul 27 21:04:17.571918 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:17.571725    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 21:04:17.718572 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:17.718539    2571 generic.go:356] "Generic (PLEG): container finished" podID="8d760a2e-48ae-4021-9766-7c875e7ec1c4" containerID="b2473a61cb7dc2c0b7448401ec719f5b5f6ba07c4b7ae8935bdb4b25cb6ed1c7" exitCode=0
Jul 27 21:04:17.719041 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:17.718606    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttt2t" event={"ID":"8d760a2e-48ae-4021-9766-7c875e7ec1c4","Type":"ContainerDied","Data":"b2473a61cb7dc2c0b7448401ec719f5b5f6ba07c4b7ae8935bdb4b25cb6ed1c7"}
Jul 27 21:04:17.799186 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:17.799158    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-tls\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:17.799321 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:17.799202    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40a3772d-f99d-45ae-bb4e-3bd833d47da0-metrics-tls\") pod \"dns-default-rc6n5\" (UID: \"40a3772d-f99d-45ae-bb4e-3bd833d47da0\") " pod="openshift-dns/dns-default-rc6n5"
Jul 27 21:04:17.799321 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:17.799313    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:04:17.799417 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:17.799329    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58ff8dd5bb-fv56s: secret "image-registry-tls" not found
Jul 27 21:04:17.799417 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:17.799335    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:04:17.799417 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:17.799382    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-tls podName:f6cf37cf-d28f-47fe-8a27-6585ab6e1579 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:19.799363438 +0000 UTC m=+39.734579800 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-tls") pod "image-registry-58ff8dd5bb-fv56s" (UID: "f6cf37cf-d28f-47fe-8a27-6585ab6e1579") : secret "image-registry-tls" not found
Jul 27 21:04:17.799417 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:17.799407    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40a3772d-f99d-45ae-bb4e-3bd833d47da0-metrics-tls podName:40a3772d-f99d-45ae-bb4e-3bd833d47da0 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:19.799398791 +0000 UTC m=+39.734615149 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/40a3772d-f99d-45ae-bb4e-3bd833d47da0-metrics-tls") pod "dns-default-rc6n5" (UID: "40a3772d-f99d-45ae-bb4e-3bd833d47da0") : secret "dns-default-metrics-tls" not found
Jul 27 21:04:18.725148 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:18.725108    2571 generic.go:356] "Generic (PLEG): container finished" podID="8d760a2e-48ae-4021-9766-7c875e7ec1c4" containerID="7444a07ab89b8f80be320d0739920de5dc07cee13fa9223e74124618f65af1f3" exitCode=0
Jul 27 21:04:18.725537 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:18.725158    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttt2t" event={"ID":"8d760a2e-48ae-4021-9766-7c875e7ec1c4","Type":"ContainerDied","Data":"7444a07ab89b8f80be320d0739920de5dc07cee13fa9223e74124618f65af1f3"}
Jul 27 21:04:19.074035 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:19.073990    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:19.140023 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:19.139918    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:19.240035 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:19.240006    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:19.340505 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:19.340478    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:19.728808 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:19.728724    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xpx7g" event={"ID":"5d872ddb-8292-4ecc-b10b-98e6c884724e","Type":"ContainerStarted","Data":"48a6017b16a74bb5c0521a018101c92f99ca33bb0c5cf40ce84dcee80b9594e8"}
Jul 27 21:04:19.732648 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:19.732623    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttt2t" event={"ID":"8d760a2e-48ae-4021-9766-7c875e7ec1c4","Type":"ContainerStarted","Data":"d0a5e83786d39e65bfcbe800552279e6c18eec5d923795866ff98561ae139a5d"}
Jul 27 21:04:19.746124 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:19.746069    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xpx7g" podStartSLOduration=2.152567451 podStartE2EDuration="4.746051787s" totalImagesPullingTime="2.593484336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:19.745446938 +0000 UTC m=+39.680663318" watchObservedRunningTime="2026-07-27 21:04:19.746051787 +0000 UTC m=+39.681268169"
Jul 27 21:04:19.770555 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:19.770429    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ttt2t" podStartSLOduration=8.216397555 podStartE2EDuration="38.770404981s" totalImagesPullingTime="30.554007426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:19.769520027 +0000 UTC m=+39.704736406" watchObservedRunningTime="2026-07-27 21:04:19.770404981 +0000 UTC m=+39.705621361"
Jul 27 21:04:19.813784 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:19.813744    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-tls\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:19.813784 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:19.813783    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40a3772d-f99d-45ae-bb4e-3bd833d47da0-metrics-tls\") pod \"dns-default-rc6n5\" (UID: \"40a3772d-f99d-45ae-bb4e-3bd833d47da0\") " pod="openshift-dns/dns-default-rc6n5"
Jul 27 21:04:19.814053 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:19.813892    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:04:19.814053 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:19.813903    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:04:19.814053 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:19.813925    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58ff8dd5bb-fv56s: secret "image-registry-tls" not found
Jul 27 21:04:19.814053 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:19.813943    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40a3772d-f99d-45ae-bb4e-3bd833d47da0-metrics-tls podName:40a3772d-f99d-45ae-bb4e-3bd833d47da0 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:23.813927602 +0000 UTC m=+43.749143958 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/40a3772d-f99d-45ae-bb4e-3bd833d47da0-metrics-tls") pod "dns-default-rc6n5" (UID: "40a3772d-f99d-45ae-bb4e-3bd833d47da0") : secret "dns-default-metrics-tls" not found
Jul 27 21:04:19.814053 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:19.814015    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-tls podName:f6cf37cf-d28f-47fe-8a27-6585ab6e1579 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:23.813995947 +0000 UTC m=+43.749212321 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-tls") pod "image-registry-58ff8dd5bb-fv56s" (UID: "f6cf37cf-d28f-47fe-8a27-6585ab6e1579") : secret "image-registry-tls" not found
Jul 27 21:04:19.891442 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:19.891402    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:19.993036 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:19.992933    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:20.293942 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:20.293908    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:20.541414 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:20.541381    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:20.634343 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:20.634249    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:20.832565 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:20.832529    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:20.901308 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:20.901096    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:21.034163 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.034121    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:21.133794 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.133764    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:21.246208 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.246124    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:21.321368 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.321335    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-hm4x2"]
Jul 27 21:04:21.345415 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.345387    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:21.354278 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.354257    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-hm4x2"]
Jul 27 21:04:21.354375 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.354353    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hm4x2"
Jul 27 21:04:21.356837 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.356816    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 27 21:04:21.356976 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.356863    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 27 21:04:21.357798 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.357778    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nb82c\""
Jul 27 21:04:21.430499 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.430451    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z896x\" (UniqueName: \"kubernetes.io/projected/012cd20d-97a2-4d3d-804b-ee2f6302bdf2-kube-api-access-z896x\") pod \"migrator-6865c9967-hm4x2\" (UID: \"012cd20d-97a2-4d3d-804b-ee2f6302bdf2\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hm4x2"
Jul 27 21:04:21.434046 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.434018    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:21.530845 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.530812    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z896x\" (UniqueName: \"kubernetes.io/projected/012cd20d-97a2-4d3d-804b-ee2f6302bdf2-kube-api-access-z896x\") pod \"migrator-6865c9967-hm4x2\" (UID: \"012cd20d-97a2-4d3d-804b-ee2f6302bdf2\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hm4x2"
Jul 27 21:04:21.541256 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.541222    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z896x\" (UniqueName: \"kubernetes.io/projected/012cd20d-97a2-4d3d-804b-ee2f6302bdf2-kube-api-access-z896x\") pod \"migrator-6865c9967-hm4x2\" (UID: \"012cd20d-97a2-4d3d-804b-ee2f6302bdf2\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hm4x2"
Jul 27 21:04:21.653415 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.653384    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:21.663015 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.662995    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hm4x2"
Jul 27 21:04:21.713431 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.713398    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:21.801267 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.801194    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-hm4x2"]
Jul 27 21:04:21.803976 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:04:21.803932    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012cd20d_97a2_4d3d_804b_ee2f6302bdf2.slice/crio-5b751b3416a7c289b5330ce117d794f39b589c761b7371156bed70ebc312ad51 WatchSource:0}: Error finding container 5b751b3416a7c289b5330ce117d794f39b589c761b7371156bed70ebc312ad51: Status 404 returned error can't find the container with id 5b751b3416a7c289b5330ce117d794f39b589c761b7371156bed70ebc312ad51
Jul 27 21:04:21.939914 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.939872    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fkz7m"]
Jul 27 21:04:21.956490 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.956442    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:21.971941 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.971910    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fkz7m"]
Jul 27 21:04:21.972097 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.972038    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:21.974886 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.974865    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 21:04:21.975028 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.974950    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 21:04:21.975104 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.975034    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 21:04:21.975188 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.975163    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w5dc2\""
Jul 27 21:04:21.975262 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:21.975246    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 21:04:22.023385 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:22.023350    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:22.034371 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:22.034346    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98e950d5-0228-4c15-b5a3-8ad6a789afa8-crio-socket\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:22.034479 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:22.034394    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98e950d5-0228-4c15-b5a3-8ad6a789afa8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:22.034479 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:22.034430    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c28wh\" (UniqueName: \"kubernetes.io/projected/98e950d5-0228-4c15-b5a3-8ad6a789afa8-kube-api-access-c28wh\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:22.034479 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:22.034471    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98e950d5-0228-4c15-b5a3-8ad6a789afa8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:22.034574 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:22.034502    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98e950d5-0228-4c15-b5a3-8ad6a789afa8-data-volume\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:22.134990 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:22.134901    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98e950d5-0228-4c15-b5a3-8ad6a789afa8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:22.134990 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:22.134940    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c28wh\" (UniqueName: \"kubernetes.io/projected/98e950d5-0228-4c15-b5a3-8ad6a789afa8-kube-api-access-c28wh\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:22.134990 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:22.134989    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98e950d5-0228-4c15-b5a3-8ad6a789afa8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:22.135164 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:22.135028    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98e950d5-0228-4c15-b5a3-8ad6a789afa8-data-volume\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:22.135164 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:22.135073    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:04:22.135164 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:22.135124    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:04:22.135164 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:22.135145    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98e950d5-0228-4c15-b5a3-8ad6a789afa8-insights-runtime-extractor-tls podName:98e950d5-0228-4c15-b5a3-8ad6a789afa8 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:22.635123673 +0000 UTC m=+42.570340050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98e950d5-0228-4c15-b5a3-8ad6a789afa8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fkz7m" (UID: "98e950d5-0228-4c15-b5a3-8ad6a789afa8") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:04:22.135313 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:22.135168    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98e950d5-0228-4c15-b5a3-8ad6a789afa8-kube-rbac-proxy-cm podName:98e950d5-0228-4c15-b5a3-8ad6a789afa8 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:22.635156191 +0000 UTC m=+42.570372549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98e950d5-0228-4c15-b5a3-8ad6a789afa8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fkz7m" (UID: "98e950d5-0228-4c15-b5a3-8ad6a789afa8") : configmap "kube-rbac-proxy" not found
Jul 27 21:04:22.135313 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:22.135079    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98e950d5-0228-4c15-b5a3-8ad6a789afa8-crio-socket\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:22.135313 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:22.135280    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98e950d5-0228-4c15-b5a3-8ad6a789afa8-crio-socket\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:22.135419 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:22.135405    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98e950d5-0228-4c15-b5a3-8ad6a789afa8-data-volume\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:22.147489 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:22.147463    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c28wh\" (UniqueName: \"kubernetes.io/projected/98e950d5-0228-4c15-b5a3-8ad6a789afa8-kube-api-access-c28wh\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:22.637911 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:22.637874    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98e950d5-0228-4c15-b5a3-8ad6a789afa8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:22.638101 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:22.638001    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98e950d5-0228-4c15-b5a3-8ad6a789afa8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:22.638101 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:22.638018    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:04:22.638101 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:22.638097    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98e950d5-0228-4c15-b5a3-8ad6a789afa8-kube-rbac-proxy-cm podName:98e950d5-0228-4c15-b5a3-8ad6a789afa8 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:23.638080396 +0000 UTC m=+43.573296753 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98e950d5-0228-4c15-b5a3-8ad6a789afa8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fkz7m" (UID: "98e950d5-0228-4c15-b5a3-8ad6a789afa8") : configmap "kube-rbac-proxy" not found
Jul 27 21:04:22.638242 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:22.638106    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:04:22.638242 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:22.638155    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98e950d5-0228-4c15-b5a3-8ad6a789afa8-insights-runtime-extractor-tls podName:98e950d5-0228-4c15-b5a3-8ad6a789afa8 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:23.638143869 +0000 UTC m=+43.573360226 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98e950d5-0228-4c15-b5a3-8ad6a789afa8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fkz7m" (UID: "98e950d5-0228-4c15-b5a3-8ad6a789afa8") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:04:22.739746 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:22.739709    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hm4x2" event={"ID":"012cd20d-97a2-4d3d-804b-ee2f6302bdf2","Type":"ContainerStarted","Data":"5b751b3416a7c289b5330ce117d794f39b589c761b7371156bed70ebc312ad51"}
Jul 27 21:04:23.524077 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.524044    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-znm99"]
Jul 27 21:04:23.539923 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.539888    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-znm99"]
Jul 27 21:04:23.540065 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.540029    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-znm99"
Jul 27 21:04:23.542546 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.542524    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 27 21:04:23.542684 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.542571    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 27 21:04:23.542855 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.542838    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 27 21:04:23.543202 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.543184    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d0e9da05-d005-4c63-90ca-9a9303f755f9-signing-cabundle\") pod \"service-ca-658d7b7c78-znm99\" (UID: \"d0e9da05-d005-4c63-90ca-9a9303f755f9\") " pod="openshift-service-ca/service-ca-658d7b7c78-znm99"
Jul 27 21:04:23.543274 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.543243    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tsss\" (UniqueName: \"kubernetes.io/projected/d0e9da05-d005-4c63-90ca-9a9303f755f9-kube-api-access-4tsss\") pod \"service-ca-658d7b7c78-znm99\" (UID: \"d0e9da05-d005-4c63-90ca-9a9303f755f9\") " pod="openshift-service-ca/service-ca-658d7b7c78-znm99"
Jul 27 21:04:23.543330 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.543313    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d0e9da05-d005-4c63-90ca-9a9303f755f9-signing-key\") pod \"service-ca-658d7b7c78-znm99\" (UID: \"d0e9da05-d005-4c63-90ca-9a9303f755f9\") " pod="openshift-service-ca/service-ca-658d7b7c78-znm99"
Jul 27 21:04:23.543461 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.543445    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 27 21:04:23.543526 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.543509    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-w4pjc\""
Jul 27 21:04:23.563096 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.563064    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:23.644310 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.644283    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d0e9da05-d005-4c63-90ca-9a9303f755f9-signing-cabundle\") pod \"service-ca-658d7b7c78-znm99\" (UID: \"d0e9da05-d005-4c63-90ca-9a9303f755f9\") " pod="openshift-service-ca/service-ca-658d7b7c78-znm99"
Jul 27 21:04:23.644452 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.644336    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98e950d5-0228-4c15-b5a3-8ad6a789afa8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:23.644452 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.644355    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tsss\" (UniqueName: \"kubernetes.io/projected/d0e9da05-d005-4c63-90ca-9a9303f755f9-kube-api-access-4tsss\") pod \"service-ca-658d7b7c78-znm99\" (UID: \"d0e9da05-d005-4c63-90ca-9a9303f755f9\") " pod="openshift-service-ca/service-ca-658d7b7c78-znm99"
Jul 27 21:04:23.644452 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.644374    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d0e9da05-d005-4c63-90ca-9a9303f755f9-signing-key\") pod \"service-ca-658d7b7c78-znm99\" (UID: \"d0e9da05-d005-4c63-90ca-9a9303f755f9\") " pod="openshift-service-ca/service-ca-658d7b7c78-znm99"
Jul 27 21:04:23.644452 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.644394    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98e950d5-0228-4c15-b5a3-8ad6a789afa8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:23.644657 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:23.644453    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:04:23.644657 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:23.644479    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:04:23.644657 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:23.644532    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98e950d5-0228-4c15-b5a3-8ad6a789afa8-insights-runtime-extractor-tls podName:98e950d5-0228-4c15-b5a3-8ad6a789afa8 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:25.644511299 +0000 UTC m=+45.579727658 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98e950d5-0228-4c15-b5a3-8ad6a789afa8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fkz7m" (UID: "98e950d5-0228-4c15-b5a3-8ad6a789afa8") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:04:23.644657 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:23.644553    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98e950d5-0228-4c15-b5a3-8ad6a789afa8-kube-rbac-proxy-cm podName:98e950d5-0228-4c15-b5a3-8ad6a789afa8 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:25.644542108 +0000 UTC m=+45.579758468 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98e950d5-0228-4c15-b5a3-8ad6a789afa8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fkz7m" (UID: "98e950d5-0228-4c15-b5a3-8ad6a789afa8") : configmap "kube-rbac-proxy" not found
Jul 27 21:04:23.644984 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.644947    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d0e9da05-d005-4c63-90ca-9a9303f755f9-signing-cabundle\") pod \"service-ca-658d7b7c78-znm99\" (UID: \"d0e9da05-d005-4c63-90ca-9a9303f755f9\") " pod="openshift-service-ca/service-ca-658d7b7c78-znm99"
Jul 27 21:04:23.646800 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.646778    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d0e9da05-d005-4c63-90ca-9a9303f755f9-signing-key\") pod \"service-ca-658d7b7c78-znm99\" (UID: \"d0e9da05-d005-4c63-90ca-9a9303f755f9\") " pod="openshift-service-ca/service-ca-658d7b7c78-znm99"
Jul 27 21:04:23.652310 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.652283    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tsss\" (UniqueName: \"kubernetes.io/projected/d0e9da05-d005-4c63-90ca-9a9303f755f9-kube-api-access-4tsss\") pod \"service-ca-658d7b7c78-znm99\" (UID: \"d0e9da05-d005-4c63-90ca-9a9303f755f9\") " pod="openshift-service-ca/service-ca-658d7b7c78-znm99"
Jul 27 21:04:23.743695 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.743603    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hm4x2" event={"ID":"012cd20d-97a2-4d3d-804b-ee2f6302bdf2","Type":"ContainerStarted","Data":"ba8f01f33fcc1d269f592eb85b39131bb6afdbadf3d54ef14e029e55591d9077"}
Jul 27 21:04:23.743695 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.743651    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hm4x2" event={"ID":"012cd20d-97a2-4d3d-804b-ee2f6302bdf2","Type":"ContainerStarted","Data":"709d61ef139443153bcda89bc776363418f2e71df6f38d9cbd1b623fae71b1e2"}
Jul 27 21:04:23.842635 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.842595    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:23.845695 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.845672    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-tls\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:23.845813 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.845713    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40a3772d-f99d-45ae-bb4e-3bd833d47da0-metrics-tls\") pod \"dns-default-rc6n5\" (UID: \"40a3772d-f99d-45ae-bb4e-3bd833d47da0\") " pod="openshift-dns/dns-default-rc6n5"
Jul 27 21:04:23.845855 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:23.845820    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:04:23.845855 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:23.845830    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:04:23.845855 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:23.845844    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58ff8dd5bb-fv56s: secret "image-registry-tls" not found
Jul 27 21:04:23.845941 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:23.845883    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-tls podName:f6cf37cf-d28f-47fe-8a27-6585ab6e1579 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:31.845867026 +0000 UTC m=+51.781083383 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-tls") pod "image-registry-58ff8dd5bb-fv56s" (UID: "f6cf37cf-d28f-47fe-8a27-6585ab6e1579") : secret "image-registry-tls" not found
Jul 27 21:04:23.845941 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:23.845898    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40a3772d-f99d-45ae-bb4e-3bd833d47da0-metrics-tls podName:40a3772d-f99d-45ae-bb4e-3bd833d47da0 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:31.845890163 +0000 UTC m=+51.781106521 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/40a3772d-f99d-45ae-bb4e-3bd833d47da0-metrics-tls") pod "dns-default-rc6n5" (UID: "40a3772d-f99d-45ae-bb4e-3bd833d47da0") : secret "dns-default-metrics-tls" not found
Jul 27 21:04:23.849637 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.849619    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-znm99"
Jul 27 21:04:23.943169 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.943137    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:23.982888 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.982835    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hm4x2" podStartSLOduration=1.371200213 podStartE2EDuration="2.982815509s" totalImagesPullingTime="1.611615296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:23.759666529 +0000 UTC m=+43.694882908" watchObservedRunningTime="2026-07-27 21:04:23.982815509 +0000 UTC m=+43.918031879"
Jul 27 21:04:23.983294 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:23.983279    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-znm99"]
Jul 27 21:04:23.986284 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:04:23.986263    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e9da05_d005_4c63_90ca_9a9303f755f9.slice/crio-365ec35a4be525a362d61b375aa14c2f56f8013aa842679afd11df527901f8fb WatchSource:0}: Error finding container 365ec35a4be525a362d61b375aa14c2f56f8013aa842679afd11df527901f8fb: Status 404 returned error can't find the container with id 365ec35a4be525a362d61b375aa14c2f56f8013aa842679afd11df527901f8fb
Jul 27 21:04:24.317107 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:24.317074    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gtpdk_c9d12f6c-3f55-44b3-b1d5-be1215730520/dns-node-resolver/0.log"
Jul 27 21:04:24.702217 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:24.702127    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:24.747800 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:24.747753    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-znm99" event={"ID":"d0e9da05-d005-4c63-90ca-9a9303f755f9","Type":"ContainerStarted","Data":"365ec35a4be525a362d61b375aa14c2f56f8013aa842679afd11df527901f8fb"}
Jul 27 21:04:25.117156 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:25.117123    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cwvnf_3de75e46-ec68-4801-9493-b1b31f67eefb/node-ca/0.log"
Jul 27 21:04:25.661781 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:25.661744    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98e950d5-0228-4c15-b5a3-8ad6a789afa8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:25.662003 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:25.661803    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98e950d5-0228-4c15-b5a3-8ad6a789afa8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:25.662003 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:25.661917    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:04:25.662003 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:25.661917    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:04:25.662307 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:25.662009    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98e950d5-0228-4c15-b5a3-8ad6a789afa8-insights-runtime-extractor-tls podName:98e950d5-0228-4c15-b5a3-8ad6a789afa8 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:29.661987692 +0000 UTC m=+49.597204052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98e950d5-0228-4c15-b5a3-8ad6a789afa8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fkz7m" (UID: "98e950d5-0228-4c15-b5a3-8ad6a789afa8") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:04:25.662307 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:25.662037    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98e950d5-0228-4c15-b5a3-8ad6a789afa8-kube-rbac-proxy-cm podName:98e950d5-0228-4c15-b5a3-8ad6a789afa8 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:29.662025941 +0000 UTC m=+49.597242298 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98e950d5-0228-4c15-b5a3-8ad6a789afa8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fkz7m" (UID: "98e950d5-0228-4c15-b5a3-8ad6a789afa8") : configmap "kube-rbac-proxy" not found
Jul 27 21:04:26.115422 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:26.115393    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xpx7g_5d872ddb-8292-4ecc-b10b-98e6c884724e/serve-healthcheck-canary/0.log"
Jul 27 21:04:26.315160 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:26.315133    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-hm4x2_012cd20d-97a2-4d3d-804b-ee2f6302bdf2/migrator/0.log"
Jul 27 21:04:26.514612 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:26.514510    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-hm4x2_012cd20d-97a2-4d3d-804b-ee2f6302bdf2/graceful-termination/0.log"
Jul 27 21:04:26.755500 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:26.755462    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-znm99" event={"ID":"d0e9da05-d005-4c63-90ca-9a9303f755f9","Type":"ContainerStarted","Data":"62eaab5f33c4fa7d9fe2337706191142d17f8523b6e3ea411f9625d7bf67a5c0"}
Jul 27 21:04:26.773636 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:26.773580    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-znm99" podStartSLOduration=1.608753594 podStartE2EDuration="3.773564952s" totalImagesPullingTime="2.164811358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:26.772487343 +0000 UTC m=+46.707703713" watchObservedRunningTime="2026-07-27 21:04:26.773564952 +0000 UTC m=+46.708781330"
Jul 27 21:04:26.959238 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:26.959201    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:27.058806 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:27.058707    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:27.328777 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:27.328690    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:27.436745 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:27.436709    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:27.521826 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:27.521790    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:27.644463 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:27.644342    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:27.906917 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:27.906827    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:28.852175 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:28.852136    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:29.045239 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:29.045201    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:29.250368 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:29.250262    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:29.453686 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:29.453632    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:29.656605 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:29.656545    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:29.696331 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:29.696301    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98e950d5-0228-4c15-b5a3-8ad6a789afa8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:29.696509 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:29.696341    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98e950d5-0228-4c15-b5a3-8ad6a789afa8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:29.696509 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:29.696428    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:04:29.696509 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:29.696498    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98e950d5-0228-4c15-b5a3-8ad6a789afa8-kube-rbac-proxy-cm podName:98e950d5-0228-4c15-b5a3-8ad6a789afa8 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:37.696471394 +0000 UTC m=+57.631687752 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98e950d5-0228-4c15-b5a3-8ad6a789afa8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fkz7m" (UID: "98e950d5-0228-4c15-b5a3-8ad6a789afa8") : configmap "kube-rbac-proxy" not found
Jul 27 21:04:29.696689 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:29.696637    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:04:29.696816 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:04:29.696735    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98e950d5-0228-4c15-b5a3-8ad6a789afa8-insights-runtime-extractor-tls podName:98e950d5-0228-4c15-b5a3-8ad6a789afa8 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:37.696715355 +0000 UTC m=+57.631931719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98e950d5-0228-4c15-b5a3-8ad6a789afa8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fkz7m" (UID: "98e950d5-0228-4c15-b5a3-8ad6a789afa8") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:04:30.649634 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:30.649589    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:30.713945 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:30.713910    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:30.882436 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:30.882390    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:31.046522 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:31.046477    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:31.245294 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:31.245263    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:31.447007 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:31.446906    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:31.546189 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:31.546154    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:31.646192 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:31.646158    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:31.915822 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:31.915784    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-tls\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:31.915822 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:31.915825    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40a3772d-f99d-45ae-bb4e-3bd833d47da0-metrics-tls\") pod \"dns-default-rc6n5\" (UID: \"40a3772d-f99d-45ae-bb4e-3bd833d47da0\") " pod="openshift-dns/dns-default-rc6n5"
Jul 27 21:04:31.918262 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:31.918239    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40a3772d-f99d-45ae-bb4e-3bd833d47da0-metrics-tls\") pod \"dns-default-rc6n5\" (UID: \"40a3772d-f99d-45ae-bb4e-3bd833d47da0\") " pod="openshift-dns/dns-default-rc6n5"
Jul 27 21:04:31.918262 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:31.918261    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-tls\") pod \"image-registry-58ff8dd5bb-fv56s\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") " pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:31.923166 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:31.923141    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rc6n5"
Jul 27 21:04:32.070110 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:32.070042    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rc6n5"]
Jul 27 21:04:32.074712 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:04:32.074688    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a3772d_f99d_45ae_bb4e_3bd833d47da0.slice/crio-1d5c49374c295c0fef05633a98cca4d5e3d87e11f6913a16026b7020effad8d0 WatchSource:0}: Error finding container 1d5c49374c295c0fef05633a98cca4d5e3d87e11f6913a16026b7020effad8d0: Status 404 returned error can't find the container with id 1d5c49374c295c0fef05633a98cca4d5e3d87e11f6913a16026b7020effad8d0
Jul 27 21:04:32.215264 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:32.215192    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:32.372631 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:32.372597    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58ff8dd5bb-fv56s"]
Jul 27 21:04:32.375393 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:04:32.375365    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6cf37cf_d28f_47fe_8a27_6585ab6e1579.slice/crio-7d6dcc9d02e8a6d1af989a0c3bb154394c129ad39b30dd0b9416725df04135e9 WatchSource:0}: Error finding container 7d6dcc9d02e8a6d1af989a0c3bb154394c129ad39b30dd0b9416725df04135e9: Status 404 returned error can't find the container with id 7d6dcc9d02e8a6d1af989a0c3bb154394c129ad39b30dd0b9416725df04135e9
Jul 27 21:04:32.671651 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:32.671616    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:32.771515 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:32.771484    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s" event={"ID":"f6cf37cf-d28f-47fe-8a27-6585ab6e1579","Type":"ContainerStarted","Data":"4055e138e5ac5196beed1fbacaae9607c261d5cc5103283dd965b3ee151f15d1"}
Jul 27 21:04:32.771713 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:32.771541    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s" event={"ID":"f6cf37cf-d28f-47fe-8a27-6585ab6e1579","Type":"ContainerStarted","Data":"7d6dcc9d02e8a6d1af989a0c3bb154394c129ad39b30dd0b9416725df04135e9"}
Jul 27 21:04:32.771713 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:32.771568    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:32.772574 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:32.772545    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rc6n5" event={"ID":"40a3772d-f99d-45ae-bb4e-3bd833d47da0","Type":"ContainerStarted","Data":"1d5c49374c295c0fef05633a98cca4d5e3d87e11f6913a16026b7020effad8d0"}
Jul 27 21:04:32.813507 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:32.813453    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s" podStartSLOduration=19.813436793 podStartE2EDuration="19.813436793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:32.812587081 +0000 UTC m=+52.747803461" watchObservedRunningTime="2026-07-27 21:04:32.813436793 +0000 UTC m=+52.748653171"
Jul 27 21:04:32.902891 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:32.902795    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:33.262642 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:33.262507    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:33.349324 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:33.349282    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:33.472404 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:33.472360    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:33.861587 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:33.861553    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:34.487549 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:34.487513    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:34.560548 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:34.560512    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:34.780749 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:34.780711    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rc6n5" event={"ID":"40a3772d-f99d-45ae-bb4e-3bd833d47da0","Type":"ContainerStarted","Data":"806c8ff5b1c9dcad63459d929b828382bfda3e7e29ef2dc9a0aa2f851afab09a"}
Jul 27 21:04:34.780932 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:34.780792    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rc6n5" event={"ID":"40a3772d-f99d-45ae-bb4e-3bd833d47da0","Type":"ContainerStarted","Data":"3f6545339e13a345a1055258646d9fff02dc54f24affb74d47dfe6749fd48614"}
Jul 27 21:04:34.780932 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:34.780863    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rc6n5"
Jul 27 21:04:34.801044 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:34.800994    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rc6n5" podStartSLOduration=17.295786008 podStartE2EDuration="19.800978028s" totalImagesPullingTime="2.50519202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:34.799379956 +0000 UTC m=+54.734596341" watchObservedRunningTime="2026-07-27 21:04:34.800978028 +0000 UTC m=+54.736194400"
Jul 27 21:04:34.845032 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:34.844998    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:35.044410 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:35.044371    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:35.105393 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:35.105355    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:35.659472 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:35.659433    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:36.257864 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:36.257828    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:36.444232 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:36.444196    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:36.586120 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:36.586085    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:36.661123 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:36.661072    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:37.047603 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:37.047577    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:37.451490 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:37.451394    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:37.652989 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:37.652934    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:37.766801 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:37.766770    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98e950d5-0228-4c15-b5a3-8ad6a789afa8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:37.767200 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:37.766819    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98e950d5-0228-4c15-b5a3-8ad6a789afa8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:37.767357 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:37.767340    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98e950d5-0228-4c15-b5a3-8ad6a789afa8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:37.769151 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:37.769123    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98e950d5-0228-4c15-b5a3-8ad6a789afa8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fkz7m\" (UID: \"98e950d5-0228-4c15-b5a3-8ad6a789afa8\") " pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:37.881566 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:37.881530    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fkz7m"
Jul 27 21:04:38.020058 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:38.019969    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fkz7m"]
Jul 27 21:04:38.024211 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:04:38.024185    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e950d5_0228_4c15_b5a3_8ad6a789afa8.slice/crio-ccdd5938c32914d7444258c068bbcba3cfb73da1e49f3d74cf4f5b9934f27cf4 WatchSource:0}: Error finding container ccdd5938c32914d7444258c068bbcba3cfb73da1e49f3d74cf4f5b9934f27cf4: Status 404 returned error can't find the container with id ccdd5938c32914d7444258c068bbcba3cfb73da1e49f3d74cf4f5b9934f27cf4
Jul 27 21:04:38.796544 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:38.796504    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fkz7m" event={"ID":"98e950d5-0228-4c15-b5a3-8ad6a789afa8","Type":"ContainerStarted","Data":"ccdd5938c32914d7444258c068bbcba3cfb73da1e49f3d74cf4f5b9934f27cf4"}
Jul 27 21:04:40.866140 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:40.866098    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:40.923400 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:40.923365    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:41.714645 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:41.714620    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkcbq"
Jul 27 21:04:43.819137 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.819061    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fkz7m" event={"ID":"98e950d5-0228-4c15-b5a3-8ad6a789afa8","Type":"ContainerStarted","Data":"df1387632a13af61098cad55c854010ceb447524a87f48d5fba0d14203c45d67"}
Jul 27 21:04:43.857633 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.856589    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-df654fc8-6lppx"]
Jul 27 21:04:43.860765 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.858761    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-79dbf8cd8-6d89v"]
Jul 27 21:04:43.860765 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.859808    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-df654fc8-6lppx"
Jul 27 21:04:43.861793 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.861778    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79dbf8cd8-6d89v"
Jul 27 21:04:43.863340 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.863159    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 27 21:04:43.866844 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.866327    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"]
Jul 27 21:04:43.866844 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.866436    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 27 21:04:43.866844 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.866450    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 27 21:04:43.866844 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.866704    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 27 21:04:43.867185 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.867131    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-m245f\""
Jul 27 21:04:43.868249 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.868230    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 27 21:04:43.870118 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.870100    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58ff8dd5bb-fv56s"]
Jul 27 21:04:43.870816 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.870240    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"
Jul 27 21:04:43.875887 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.875842    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 27 21:04:43.875887 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.875880    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 27 21:04:43.876321 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.876160    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 27 21:04:43.877229 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.877212    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-df654fc8-6lppx"]
Jul 27 21:04:43.878630 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.878137    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 27 21:04:43.887804 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.887464    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-79dbf8cd8-6d89v"]
Jul 27 21:04:43.888822 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.888780    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"]
Jul 27 21:04:43.914309 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.914034    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3a33373f-f9d0-42c0-8a68-de018145c1a5-klusterlet-config\") pod \"klusterlet-addon-workmgr-79dbf8cd8-6d89v\" (UID: \"3a33373f-f9d0-42c0-8a68-de018145c1a5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79dbf8cd8-6d89v"
Jul 27 21:04:43.914309 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.914082    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp45m\" (UniqueName: \"kubernetes.io/projected/1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0-kube-api-access-zp45m\") pod \"cluster-proxy-proxy-agent-77c58b6744-7gkhp\" (UID: \"1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"
Jul 27 21:04:43.914309 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.914114    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77c58b6744-7gkhp\" (UID: \"1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"
Jul 27 21:04:43.914309 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.914156    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0-ca\") pod \"cluster-proxy-proxy-agent-77c58b6744-7gkhp\" (UID: \"1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"
Jul 27 21:04:43.914309 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.914182    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77c58b6744-7gkhp\" (UID: \"1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"
Jul 27 21:04:43.914309 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.914241    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d0ed2ea8-bff1-424b-94bc-cc73570d64e1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-df654fc8-6lppx\" (UID: \"d0ed2ea8-bff1-424b-94bc-cc73570d64e1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-df654fc8-6lppx"
Jul 27 21:04:43.914631 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.914356    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77c58b6744-7gkhp\" (UID: \"1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"
Jul 27 21:04:43.914631 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.914380    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0-hub\") pod \"cluster-proxy-proxy-agent-77c58b6744-7gkhp\" (UID: \"1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"
Jul 27 21:04:43.914631 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.914402    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnffr\" (UniqueName: \"kubernetes.io/projected/3a33373f-f9d0-42c0-8a68-de018145c1a5-kube-api-access-hnffr\") pod \"klusterlet-addon-workmgr-79dbf8cd8-6d89v\" (UID: \"3a33373f-f9d0-42c0-8a68-de018145c1a5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79dbf8cd8-6d89v"
Jul 27 21:04:43.914631 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.914443    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a33373f-f9d0-42c0-8a68-de018145c1a5-tmp\") pod \"klusterlet-addon-workmgr-79dbf8cd8-6d89v\" (UID: \"3a33373f-f9d0-42c0-8a68-de018145c1a5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79dbf8cd8-6d89v"
Jul 27 21:04:43.914631 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.914515    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjzvv\" (UniqueName: \"kubernetes.io/projected/d0ed2ea8-bff1-424b-94bc-cc73570d64e1-kube-api-access-cjzvv\") pod \"managed-serviceaccount-addon-agent-df654fc8-6lppx\" (UID: \"d0ed2ea8-bff1-424b-94bc-cc73570d64e1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-df654fc8-6lppx"
Jul 27 21:04:43.961500 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.961471    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-89d6577f8-nz7f2"]
Jul 27 21:04:43.963814 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.963789    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:43.985556 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.985511    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:43.986434 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:43.986323    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-89d6577f8-nz7f2"]
Jul 27 21:04:44.015402 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.015366    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnffr\" (UniqueName: \"kubernetes.io/projected/3a33373f-f9d0-42c0-8a68-de018145c1a5-kube-api-access-hnffr\") pod \"klusterlet-addon-workmgr-79dbf8cd8-6d89v\" (UID: \"3a33373f-f9d0-42c0-8a68-de018145c1a5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79dbf8cd8-6d89v"
Jul 27 21:04:44.015579 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.015431    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a33373f-f9d0-42c0-8a68-de018145c1a5-tmp\") pod \"klusterlet-addon-workmgr-79dbf8cd8-6d89v\" (UID: \"3a33373f-f9d0-42c0-8a68-de018145c1a5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79dbf8cd8-6d89v"
Jul 27 21:04:44.015579 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.015458    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/883d84e7-a560-4d61-92f9-7f98d8caead0-registry-certificates\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.015579 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.015477    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjzvv\" (UniqueName: \"kubernetes.io/projected/d0ed2ea8-bff1-424b-94bc-cc73570d64e1-kube-api-access-cjzvv\") pod \"managed-serviceaccount-addon-agent-df654fc8-6lppx\" (UID: \"d0ed2ea8-bff1-424b-94bc-cc73570d64e1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-df654fc8-6lppx"
Jul 27 21:04:44.015579 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.015501    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/883d84e7-a560-4d61-92f9-7f98d8caead0-ca-trust-extracted\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.015579 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.015525    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/883d84e7-a560-4d61-92f9-7f98d8caead0-installation-pull-secrets\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.015579 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.015552    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/883d84e7-a560-4d61-92f9-7f98d8caead0-trusted-ca\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.015899 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.015600    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3a33373f-f9d0-42c0-8a68-de018145c1a5-klusterlet-config\") pod \"klusterlet-addon-workmgr-79dbf8cd8-6d89v\" (UID: \"3a33373f-f9d0-42c0-8a68-de018145c1a5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79dbf8cd8-6d89v"
Jul 27 21:04:44.015899 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.015634    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp45m\" (UniqueName: \"kubernetes.io/projected/1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0-kube-api-access-zp45m\") pod \"cluster-proxy-proxy-agent-77c58b6744-7gkhp\" (UID: \"1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"
Jul 27 21:04:44.015899 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.015662    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/883d84e7-a560-4d61-92f9-7f98d8caead0-registry-tls\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.015899 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.015706    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77c58b6744-7gkhp\" (UID: \"1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"
Jul 27 21:04:44.016226 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.016206    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a33373f-f9d0-42c0-8a68-de018145c1a5-tmp\") pod \"klusterlet-addon-workmgr-79dbf8cd8-6d89v\" (UID: \"3a33373f-f9d0-42c0-8a68-de018145c1a5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79dbf8cd8-6d89v"
Jul 27 21:04:44.017751 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.016402    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0-ca\") pod \"cluster-proxy-proxy-agent-77c58b6744-7gkhp\" (UID: \"1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"
Jul 27 21:04:44.017751 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.016610    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77c58b6744-7gkhp\" (UID: \"1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"
Jul 27 21:04:44.017751 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.016644    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/883d84e7-a560-4d61-92f9-7f98d8caead0-bound-sa-token\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.017751 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.016689    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d0ed2ea8-bff1-424b-94bc-cc73570d64e1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-df654fc8-6lppx\" (UID: \"d0ed2ea8-bff1-424b-94bc-cc73570d64e1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-df654fc8-6lppx"
Jul 27 21:04:44.017751 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.016717    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/883d84e7-a560-4d61-92f9-7f98d8caead0-image-registry-private-configuration\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.017751 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.016761    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77c58b6744-7gkhp\" (UID: \"1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"
Jul 27 21:04:44.017751 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.016803    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0-hub\") pod \"cluster-proxy-proxy-agent-77c58b6744-7gkhp\" (UID: \"1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"
Jul 27 21:04:44.017751 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.016859    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctgm8\" (UniqueName: \"kubernetes.io/projected/883d84e7-a560-4d61-92f9-7f98d8caead0-kube-api-access-ctgm8\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.018767 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.018723    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77c58b6744-7gkhp\" (UID: \"1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"
Jul 27 21:04:44.018865 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.018800    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77c58b6744-7gkhp\" (UID: \"1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"
Jul 27 21:04:44.019969 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.019927    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3a33373f-f9d0-42c0-8a68-de018145c1a5-klusterlet-config\") pod \"klusterlet-addon-workmgr-79dbf8cd8-6d89v\" (UID: \"3a33373f-f9d0-42c0-8a68-de018145c1a5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79dbf8cd8-6d89v"
Jul 27 21:04:44.020717 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.020694    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0-ca\") pod \"cluster-proxy-proxy-agent-77c58b6744-7gkhp\" (UID: \"1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"
Jul 27 21:04:44.021932 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.021893    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d0ed2ea8-bff1-424b-94bc-cc73570d64e1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-df654fc8-6lppx\" (UID: \"d0ed2ea8-bff1-424b-94bc-cc73570d64e1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-df654fc8-6lppx"
Jul 27 21:04:44.022044 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.021997    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0-hub\") pod \"cluster-proxy-proxy-agent-77c58b6744-7gkhp\" (UID: \"1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"
Jul 27 21:04:44.028770 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.028717    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp45m\" (UniqueName: \"kubernetes.io/projected/1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0-kube-api-access-zp45m\") pod \"cluster-proxy-proxy-agent-77c58b6744-7gkhp\" (UID: \"1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"
Jul 27 21:04:44.029534 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.029507    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77c58b6744-7gkhp\" (UID: \"1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"
Jul 27 21:04:44.030662 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.030638    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjzvv\" (UniqueName: \"kubernetes.io/projected/d0ed2ea8-bff1-424b-94bc-cc73570d64e1-kube-api-access-cjzvv\") pod \"managed-serviceaccount-addon-agent-df654fc8-6lppx\" (UID: \"d0ed2ea8-bff1-424b-94bc-cc73570d64e1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-df654fc8-6lppx"
Jul 27 21:04:44.032441 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.032420    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnffr\" (UniqueName: \"kubernetes.io/projected/3a33373f-f9d0-42c0-8a68-de018145c1a5-kube-api-access-hnffr\") pod \"klusterlet-addon-workmgr-79dbf8cd8-6d89v\" (UID: \"3a33373f-f9d0-42c0-8a68-de018145c1a5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79dbf8cd8-6d89v"
Jul 27 21:04:44.076361 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.076290    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:44.118293 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.118258    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/883d84e7-a560-4d61-92f9-7f98d8caead0-installation-pull-secrets\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.118440 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.118303    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/883d84e7-a560-4d61-92f9-7f98d8caead0-trusted-ca\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.118440 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.118363    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/883d84e7-a560-4d61-92f9-7f98d8caead0-registry-tls\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.118440 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.118413    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/883d84e7-a560-4d61-92f9-7f98d8caead0-bound-sa-token\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.118615 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.118441    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/883d84e7-a560-4d61-92f9-7f98d8caead0-image-registry-private-configuration\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.118615 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.118503    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctgm8\" (UniqueName: \"kubernetes.io/projected/883d84e7-a560-4d61-92f9-7f98d8caead0-kube-api-access-ctgm8\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.118615 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.118549    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/883d84e7-a560-4d61-92f9-7f98d8caead0-registry-certificates\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.118615 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.118578    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/883d84e7-a560-4d61-92f9-7f98d8caead0-ca-trust-extracted\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.119797 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.119766    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/883d84e7-a560-4d61-92f9-7f98d8caead0-trusted-ca\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.120107 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.120087    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/883d84e7-a560-4d61-92f9-7f98d8caead0-ca-trust-extracted\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.120731 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.120692    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/883d84e7-a560-4d61-92f9-7f98d8caead0-registry-certificates\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.121362 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.121314    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/883d84e7-a560-4d61-92f9-7f98d8caead0-image-registry-private-configuration\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.121499 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.121471    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/883d84e7-a560-4d61-92f9-7f98d8caead0-registry-tls\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.121887 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.121866    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/883d84e7-a560-4d61-92f9-7f98d8caead0-installation-pull-secrets\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.130276 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.130255    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctgm8\" (UniqueName: \"kubernetes.io/projected/883d84e7-a560-4d61-92f9-7f98d8caead0-kube-api-access-ctgm8\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.134018 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.133995    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/883d84e7-a560-4d61-92f9-7f98d8caead0-bound-sa-token\") pod \"image-registry-89d6577f8-nz7f2\" (UID: \"883d84e7-a560-4d61-92f9-7f98d8caead0\") " pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.185513 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.185478    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:44.188186 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.188166    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-df654fc8-6lppx"
Jul 27 21:04:44.199735 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.199716    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79dbf8cd8-6d89v"
Jul 27 21:04:44.206157 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.205942    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"
Jul 27 21:04:44.290496 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.289078    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.290496 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.289859    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:44.373022 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.372942    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-df654fc8-6lppx"]
Jul 27 21:04:44.380037 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:04:44.380008    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0ed2ea8_bff1_424b_94bc_cc73570d64e1.slice/crio-7fcaa26fcdf1af0e924ee2f5ce49da47de55d82bee41a22a51d17512064b49ae WatchSource:0}: Error finding container 7fcaa26fcdf1af0e924ee2f5ce49da47de55d82bee41a22a51d17512064b49ae: Status 404 returned error can't find the container with id 7fcaa26fcdf1af0e924ee2f5ce49da47de55d82bee41a22a51d17512064b49ae
Jul 27 21:04:44.393004 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.392974    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-79dbf8cd8-6d89v"]
Jul 27 21:04:44.396818 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:04:44.396785    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a33373f_f9d0_42c0_8a68_de018145c1a5.slice/crio-e3ba5ec64ea109a1338179130f7df3db0fec1d24238c0ebd0b15c2e6e4a5da5a WatchSource:0}: Error finding container e3ba5ec64ea109a1338179130f7df3db0fec1d24238c0ebd0b15c2e6e4a5da5a: Status 404 returned error can't find the container with id e3ba5ec64ea109a1338179130f7df3db0fec1d24238c0ebd0b15c2e6e4a5da5a
Jul 27 21:04:44.398032 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.397894    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp"]
Jul 27 21:04:44.400777 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:04:44.400755    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d7384ed_406f_4bc9_8b7f_acbb73f3f7c0.slice/crio-7ae2e56ac3dda15fe514234be312eb56f77d3833ec6c24d65ff6f1d58ee8aef8 WatchSource:0}: Error finding container 7ae2e56ac3dda15fe514234be312eb56f77d3833ec6c24d65ff6f1d58ee8aef8: Status 404 returned error can't find the container with id 7ae2e56ac3dda15fe514234be312eb56f77d3833ec6c24d65ff6f1d58ee8aef8
Jul 27 21:04:44.473893 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:04:44.473863    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883d84e7_a560_4d61_92f9_7f98d8caead0.slice/crio-e9c975b13de95bbc613fa703987e29f75c93a5bef36336ebfbed6ac84b8e8209 WatchSource:0}: Error finding container e9c975b13de95bbc613fa703987e29f75c93a5bef36336ebfbed6ac84b8e8209: Status 404 returned error can't find the container with id e9c975b13de95bbc613fa703987e29f75c93a5bef36336ebfbed6ac84b8e8209
Jul 27 21:04:44.476143 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.476122    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-89d6577f8-nz7f2"]
Jul 27 21:04:44.562775 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.562742    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:44.655254 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.655123    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:44.786545 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.786513    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rc6n5"
Jul 27 21:04:44.828763 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.828668    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:44.829370 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.829345    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp" event={"ID":"1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0","Type":"ContainerStarted","Data":"7ae2e56ac3dda15fe514234be312eb56f77d3833ec6c24d65ff6f1d58ee8aef8"}
Jul 27 21:04:44.831132 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.831094    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-89d6577f8-nz7f2" event={"ID":"883d84e7-a560-4d61-92f9-7f98d8caead0","Type":"ContainerStarted","Data":"b3951585542db9940cae81a9b197f5b1a08b59ca0a7f12827391aa62bddcda32"}
Jul 27 21:04:44.831253 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.831147    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-89d6577f8-nz7f2" event={"ID":"883d84e7-a560-4d61-92f9-7f98d8caead0","Type":"ContainerStarted","Data":"e9c975b13de95bbc613fa703987e29f75c93a5bef36336ebfbed6ac84b8e8209"}
Jul 27 21:04:44.831865 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.831838    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:04:44.833312 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.833270    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79dbf8cd8-6d89v" event={"ID":"3a33373f-f9d0-42c0-8a68-de018145c1a5","Type":"ContainerStarted","Data":"e3ba5ec64ea109a1338179130f7df3db0fec1d24238c0ebd0b15c2e6e4a5da5a"}
Jul 27 21:04:44.834622 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.834597    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-df654fc8-6lppx" event={"ID":"d0ed2ea8-bff1-424b-94bc-cc73570d64e1","Type":"ContainerStarted","Data":"7fcaa26fcdf1af0e924ee2f5ce49da47de55d82bee41a22a51d17512064b49ae"}
Jul 27 21:04:44.858731 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.858665    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-89d6577f8-nz7f2" podStartSLOduration=1.858644067 podStartE2EDuration="1.858644067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:44.855746607 +0000 UTC m=+64.790962986" watchObservedRunningTime="2026-07-27 21:04:44.858644067 +0000 UTC m=+64.793860447"
Jul 27 21:04:44.894863 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:44.894806    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:46.239726 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:46.239682    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret\") pod \"global-pull-secret-syncer-kvpv2\" (UID: \"71b80e11-ddee-4d86-a0b9-632732d0cca2\") " pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:04:46.240208 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:46.239770    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs\") pod \"network-metrics-daemon-whppm\" (UID: \"222ea6ea-61b8-4854-bc88-834449ce2974\") " pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:04:46.240208 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:46.239825    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4rgh\" (UniqueName: \"kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh\") pod \"network-check-target-m2bn8\" (UID: \"c268e820-791d-448a-b5ec-83ec69ad08ee\") " pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:04:46.242826 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:46.242797    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 21:04:46.243769 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:46.243476    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 21:04:46.243769 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:46.243705    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 21:04:46.254092 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:46.254065    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 21:04:46.259378 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:46.259352    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/222ea6ea-61b8-4854-bc88-834449ce2974-metrics-certs\") pod \"network-metrics-daemon-whppm\" (UID: \"222ea6ea-61b8-4854-bc88-834449ce2974\") " pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:04:46.263126 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:46.263074    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71b80e11-ddee-4d86-a0b9-632732d0cca2-original-pull-secret\") pod \"global-pull-secret-syncer-kvpv2\" (UID: \"71b80e11-ddee-4d86-a0b9-632732d0cca2\") " pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:04:46.268287 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:46.268252    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4rgh\" (UniqueName: \"kubernetes.io/projected/c268e820-791d-448a-b5ec-83ec69ad08ee-kube-api-access-v4rgh\") pod \"network-check-target-m2bn8\" (UID: \"c268e820-791d-448a-b5ec-83ec69ad08ee\") " pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:04:46.380452 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:46.380418    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bztw5\""
Jul 27 21:04:46.382885 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:46.382673    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kvpv2"
Jul 27 21:04:46.388396 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:46.388331    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-whppm"
Jul 27 21:04:46.390862 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:46.390838    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ww7jp\""
Jul 27 21:04:46.399038 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:46.399018    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:04:46.961601 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:46.961570    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:47.061100 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:47.060944    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:47.201283 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:47.201232    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:47.275480 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:47.275430    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:49.187263 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:49.187213    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:49.286044 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:49.286011    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:49.382487 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:49.382451    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:50.171400 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:50.171365    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:50.271184 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:50.271154    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:50.412388 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:50.411987    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-whppm"]
Jul 27 21:04:50.437232 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:50.437164    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m2bn8"]
Jul 27 21:04:51.471488 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:04:51.471100    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222ea6ea_61b8_4854_bc88_834449ce2974.slice/crio-3935d550e310fbc1be5af3f77d69bf09c0033938015bd5ec181c88cb024dc6e4 WatchSource:0}: Error finding container 3935d550e310fbc1be5af3f77d69bf09c0033938015bd5ec181c88cb024dc6e4: Status 404 returned error can't find the container with id 3935d550e310fbc1be5af3f77d69bf09c0033938015bd5ec181c88cb024dc6e4
Jul 27 21:04:51.604021 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:51.603994    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kvpv2"]
Jul 27 21:04:51.609220 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:04:51.609190    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b80e11_ddee_4d86_a0b9_632732d0cca2.slice/crio-03c5f1b91c34835e0c6f4a020014d5fe52e362682a958dbcd75dbde790c4389a WatchSource:0}: Error finding container 03c5f1b91c34835e0c6f4a020014d5fe52e362682a958dbcd75dbde790c4389a: Status 404 returned error can't find the container with id 03c5f1b91c34835e0c6f4a020014d5fe52e362682a958dbcd75dbde790c4389a
Jul 27 21:04:51.862939 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:51.862899    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m2bn8" event={"ID":"c268e820-791d-448a-b5ec-83ec69ad08ee","Type":"ContainerStarted","Data":"827ab3163982a88fe1d15729af1e1ac3e4e44ffef2e0fe22eea5f57058e796f7"}
Jul 27 21:04:51.864130 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:51.864102    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-whppm" event={"ID":"222ea6ea-61b8-4854-bc88-834449ce2974","Type":"ContainerStarted","Data":"3935d550e310fbc1be5af3f77d69bf09c0033938015bd5ec181c88cb024dc6e4"}
Jul 27 21:04:51.866178 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:51.866151    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fkz7m" event={"ID":"98e950d5-0228-4c15-b5a3-8ad6a789afa8","Type":"ContainerStarted","Data":"0af35b4d2d3029b59f4c15ed430b955e8ada859c1e9abd4bdb2c0c9596a91398"}
Jul 27 21:04:51.868400 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:51.868378    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp" event={"ID":"1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0","Type":"ContainerStarted","Data":"0c999566b394863c30186a3a9b9b9a0f6983fae0cb81c0c181af35676127cd6a"}
Jul 27 21:04:51.868502 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:51.868419    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp" event={"ID":"1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0","Type":"ContainerStarted","Data":"a1f211e6962ecddef0a0e61dd16ad22308eba10f21ccbbf7fb6fa6878b439982"}
Jul 27 21:04:51.868502 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:51.868437    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp" event={"ID":"1d7384ed-406f-4bc9-8b7f-acbb73f3f7c0","Type":"ContainerStarted","Data":"301e4eb16cc03ab9403cf9962c3cb45183ec8d1b9f421db239decf9e9b862ec3"}
Jul 27 21:04:51.870057 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:51.870034    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79dbf8cd8-6d89v" event={"ID":"3a33373f-f9d0-42c0-8a68-de018145c1a5","Type":"ContainerStarted","Data":"9830cf0dd90535018d1b1a730068800cda6643b83b5ffb49592c294272a4aea1"}
Jul 27 21:04:51.870191 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:51.870172    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79dbf8cd8-6d89v"
Jul 27 21:04:51.872102 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:51.872071    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79dbf8cd8-6d89v"
Jul 27 21:04:51.872200 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:51.872167    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-df654fc8-6lppx" event={"ID":"d0ed2ea8-bff1-424b-94bc-cc73570d64e1","Type":"ContainerStarted","Data":"4b68985a166ee0ff48d756f0b25ee62c2a66299309950b03fcbb715199e327e2"}
Jul 27 21:04:51.873499 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:51.873473    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kvpv2" event={"ID":"71b80e11-ddee-4d86-a0b9-632732d0cca2","Type":"ContainerStarted","Data":"03c5f1b91c34835e0c6f4a020014d5fe52e362682a958dbcd75dbde790c4389a"}
Jul 27 21:04:51.888505 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:51.887929    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77c58b6744-7gkhp" podStartSLOduration=1.788427231 podStartE2EDuration="8.887907792s" totalImagesPullingTime="7.099480561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:51.886361832 +0000 UTC m=+71.821578214" watchObservedRunningTime="2026-07-27 21:04:51.887907792 +0000 UTC m=+71.823124152"
Jul 27 21:04:51.926707 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:51.926645    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-df654fc8-6lppx" podStartSLOduration=1.8386028460000001 podStartE2EDuration="8.926622979s" totalImagesPullingTime="7.088020133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:51.902135174 +0000 UTC m=+71.837351554" watchObservedRunningTime="2026-07-27 21:04:51.926622979 +0000 UTC m=+71.861839359"
Jul 27 21:04:51.927180 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:51.927139    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79dbf8cd8-6d89v" podStartSLOduration=1.80501131 podStartE2EDuration="8.927126979s" totalImagesPullingTime="7.122115669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:51.919238907 +0000 UTC m=+71.854455289" watchObservedRunningTime="2026-07-27 21:04:51.927126979 +0000 UTC m=+71.862343361"
Jul 27 21:04:52.881620 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:52.881587    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-whppm" event={"ID":"222ea6ea-61b8-4854-bc88-834449ce2974","Type":"ContainerStarted","Data":"249f439c4000576034318ddd63c2ef4ee424e6e532ced51139b814e5298d5286"}
Jul 27 21:04:53.878607 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:53.878373    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:04:53.888774 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:53.888745    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-whppm" event={"ID":"222ea6ea-61b8-4854-bc88-834449ce2974","Type":"ContainerStarted","Data":"7b5bbcf592bca83d633a772880bb7443ba6d74f996bac1ef9baedace7564597f"}
Jul 27 21:04:53.921525 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:53.920938    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-whppm" podStartSLOduration=71.753710551 podStartE2EDuration="1m12.920918297s" totalImagesPullingTime="1.167207746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:53.919605296 +0000 UTC m=+73.854821676" watchObservedRunningTime="2026-07-27 21:04:53.920918297 +0000 UTC m=+73.856134676"
Jul 27 21:04:53.932220 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:53.932166    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:54.039780 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:54.039744    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:54.103202 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:54.103138    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:54.398581 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:54.398521    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:54.481520 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:54.481414    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:54.710606 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:54.710412    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:54.770774 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:54.770735    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:56.149542 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:56.149501    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:56.251394 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:56.251355    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:58.909553 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:58.909519    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kvpv2" event={"ID":"71b80e11-ddee-4d86-a0b9-632732d0cca2","Type":"ContainerStarted","Data":"cefcbab0b94d916f6772e454dc4847e55e8e224c625f60d1e459a634b76b11c0"}
Jul 27 21:04:58.910991 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:58.910939    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m2bn8" event={"ID":"c268e820-791d-448a-b5ec-83ec69ad08ee","Type":"ContainerStarted","Data":"3c4730b351a3c202a824872737005b0af3945e3c5a17fb2824e95ba657e5630b"}
Jul 27 21:04:58.911123 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:58.911066    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:04:58.912945 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:58.912925    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fkz7m" event={"ID":"98e950d5-0228-4c15-b5a3-8ad6a789afa8","Type":"ContainerStarted","Data":"cd442546570ca6d3b072d926a678b13368fea883c08c2ae1f405494469679eb7"}
Jul 27 21:04:58.949878 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:58.949812    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kvpv2" podStartSLOduration=71.508265967 podStartE2EDuration="1m17.949798332s" totalImagesPullingTime="6.441532365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:58.928016288 +0000 UTC m=+78.863232668" watchObservedRunningTime="2026-07-27 21:04:58.949798332 +0000 UTC m=+78.885014711"
Jul 27 21:04:58.950377 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:58.950334    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m2bn8" podStartSLOduration=71.38661607 podStartE2EDuration="1m17.950324742s" totalImagesPullingTime="6.563708672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:58.950120621 +0000 UTC m=+78.885337000" watchObservedRunningTime="2026-07-27 21:04:58.950324742 +0000 UTC m=+78.885541122"
Jul 27 21:04:58.975172 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:58.975138    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fkz7m" podStartSLOduration=18.150181632 podStartE2EDuration="37.975125385s" totalImagesPullingTime="19.824943753s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:58.973541884 +0000 UTC m=+78.908758276" watchObservedRunningTime="2026-07-27 21:04:58.975125385 +0000 UTC m=+78.910341764"
Jul 27 21:04:58.987912 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:58.987886    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:59.092933 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:59.092902    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:59.400487 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:04:59.400451    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:01.557717 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:01.557685    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:01.738612 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:01.738573    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:01.838368 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:01.838285    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:01.846139 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:01.846117    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p8swf"]
Jul 27 21:05:01.849087 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:01.849064    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:01.851559 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:01.851531    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 21:05:01.851664 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:01.851629    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 21:05:01.852015 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:01.851995    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 21:05:01.852106 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:01.852028    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 21:05:01.852710 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:01.852694    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 21:05:01.852809 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:01.852791    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ntxzb\""
Jul 27 21:05:01.852914 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:01.852897    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 21:05:01.988473 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:01.988355    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-metrics-client-ca\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:01.988473 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:01.988406    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-node-exporter-textfile\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:01.988473 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:01.988433    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:01.988473 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:01.988457    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxnms\" (UniqueName: \"kubernetes.io/projected/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-kube-api-access-hxnms\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:01.988780 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:01.988489    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-root\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:01.988780 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:01.988590    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-node-exporter-accelerators-collector-config\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:01.988780 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:01.988631    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-node-exporter-tls\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:01.988780 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:01.988701    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-sys\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:01.988780 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:01.988755    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-node-exporter-wtmp\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:02.067686 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.067653    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:02.089596 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.089508    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-node-exporter-tls\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:02.089596 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.089562    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-sys\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:02.089596 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.089596    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-node-exporter-wtmp\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:02.089846 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.089639    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-metrics-client-ca\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:02.089846 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.089663    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-node-exporter-textfile\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:02.089846 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.089691    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:02.089846 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.089697    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-sys\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:02.089846 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.089713    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxnms\" (UniqueName: \"kubernetes.io/projected/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-kube-api-access-hxnms\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:02.089846 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.089767    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-root\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:02.089846 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.089805    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-node-exporter-accelerators-collector-config\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:02.090399 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.089854    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-node-exporter-wtmp\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:02.090399 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.089905    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-root\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:02.090399 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.090195    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-node-exporter-textfile\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:02.090399 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.090300    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-metrics-client-ca\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:02.090399 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.090364    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-node-exporter-accelerators-collector-config\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:02.092436 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.092415    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-node-exporter-tls\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:02.092712 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.092694    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:02.098634 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.098617    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxnms\" (UniqueName: \"kubernetes.io/projected/f1f4ab75-412a-48f7-a10b-440ee3bb02e4-kube-api-access-hxnms\") pod \"node-exporter-p8swf\" (UID: \"f1f4ab75-412a-48f7-a10b-440ee3bb02e4\") " pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:02.147357 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.147321    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:02.161145 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.161118    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p8swf"
Jul 27 21:05:02.175002 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:05:02.174974    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f4ab75_412a_48f7_a10b_440ee3bb02e4.slice/crio-0746708df350f3c7b96a8d504bca19303ce4c185e06ccee62d8540220e3da963 WatchSource:0}: Error finding container 0746708df350f3c7b96a8d504bca19303ce4c185e06ccee62d8540220e3da963: Status 404 returned error can't find the container with id 0746708df350f3c7b96a8d504bca19303ce4c185e06ccee62d8540220e3da963
Jul 27 21:05:02.271878 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.271840    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:02.373740 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.373644    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:02.477141 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.477104    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:02.582261 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.582232    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:02.678557 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.678467    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:02.795455 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.795416    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:02.886343 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.886303    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:02.927435 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:02.927399    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p8swf" event={"ID":"f1f4ab75-412a-48f7-a10b-440ee3bb02e4","Type":"ContainerStarted","Data":"0746708df350f3c7b96a8d504bca19303ce4c185e06ccee62d8540220e3da963"}
Jul 27 21:05:03.008213 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:03.008122    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:03.800203 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:03.800164    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:03.864118 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:03.864055    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:03.933343 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:03.933308    2571 generic.go:356] "Generic (PLEG): container finished" podID="f1f4ab75-412a-48f7-a10b-440ee3bb02e4" containerID="4f70bea13e3da65bd451c4e3a401e8125644c51572b173dca48553211b832c4d" exitCode=0
Jul 27 21:05:03.933578 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:03.933348    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p8swf" event={"ID":"f1f4ab75-412a-48f7-a10b-440ee3bb02e4","Type":"ContainerDied","Data":"4f70bea13e3da65bd451c4e3a401e8125644c51572b173dca48553211b832c4d"}
Jul 27 21:05:04.755921 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:04.755883    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:04.823824 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:04.823785    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:04.939119 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:04.939082    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p8swf" event={"ID":"f1f4ab75-412a-48f7-a10b-440ee3bb02e4","Type":"ContainerStarted","Data":"9b5212d36dd4b3f8f0310f4ba95394ce3c7ce569ad4e688ddc18e955ca43521b"}
Jul 27 21:05:04.939277 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:04.939136    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p8swf" event={"ID":"f1f4ab75-412a-48f7-a10b-440ee3bb02e4","Type":"ContainerStarted","Data":"72338492a719936518414e7dbcf9d38d41e40edc985488d41576df71d6d3ea50"}
Jul 27 21:05:04.955233 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:04.955204    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:04.962525 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:04.962475    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p8swf" podStartSLOduration=2.967355596 podStartE2EDuration="3.962456483s" totalImagesPullingTime="924.622171ms" totalInitContainerRuntime="70.478716ms" isStatefulPod=true podCreationTimestamp="2026-07-27 21:05:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:04.960602197 +0000 UTC m=+84.895818599" watchObservedRunningTime="2026-07-27 21:05:04.962456483 +0000 UTC m=+84.897672866"
Jul 27 21:05:05.152920 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:05.152887    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:05.410471 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:05.410389    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:05.508811 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:05.508779    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:06.570087 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:06.570051    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:06.638461 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:06.638424    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:06.848463 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:06.848392    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-89d6577f8-nz7f2"
Jul 27 21:05:06.978045 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:06.977984    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:07.094130 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:07.094106    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:07.156996 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:07.156905    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:08.087144 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:08.087085    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:08.899070 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:08.899019    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s" podUID="f6cf37cf-d28f-47fe-8a27-6585ab6e1579" containerName="registry" containerID="cri-o://4055e138e5ac5196beed1fbacaae9607c261d5cc5103283dd965b3ee151f15d1" gracePeriod=30
Jul 27 21:05:09.132719 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.132698    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:05:09.147935 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.147914    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-ca-trust-extracted\") pod \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") "
Jul 27 21:05:09.148074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.147940    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-certificates\") pod \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") "
Jul 27 21:05:09.148074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.147977    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-installation-pull-secrets\") pod \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") "
Jul 27 21:05:09.148074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.148006    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-bound-sa-token\") pod \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") "
Jul 27 21:05:09.148238 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.148144    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgcfg\" (UniqueName: \"kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-kube-api-access-vgcfg\") pod \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") "
Jul 27 21:05:09.148238 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.148206    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-image-registry-private-configuration\") pod \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") "
Jul 27 21:05:09.148238 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.148232    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-tls\") pod \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") "
Jul 27 21:05:09.148388 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.148274    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-trusted-ca\") pod \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\" (UID: \"f6cf37cf-d28f-47fe-8a27-6585ab6e1579\") "
Jul 27 21:05:09.148388 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.148345    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f6cf37cf-d28f-47fe-8a27-6585ab6e1579" (UID: "f6cf37cf-d28f-47fe-8a27-6585ab6e1579"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:05:09.148517 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.148497    2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-certificates\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:05:09.148886 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.148815    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f6cf37cf-d28f-47fe-8a27-6585ab6e1579" (UID: "f6cf37cf-d28f-47fe-8a27-6585ab6e1579"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:05:09.150705 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.150632    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f6cf37cf-d28f-47fe-8a27-6585ab6e1579" (UID: "f6cf37cf-d28f-47fe-8a27-6585ab6e1579"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:05:09.151055 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.151028    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-kube-api-access-vgcfg" (OuterVolumeSpecName: "kube-api-access-vgcfg") pod "f6cf37cf-d28f-47fe-8a27-6585ab6e1579" (UID: "f6cf37cf-d28f-47fe-8a27-6585ab6e1579"). InnerVolumeSpecName "kube-api-access-vgcfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:05:09.151168 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.151075    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f6cf37cf-d28f-47fe-8a27-6585ab6e1579" (UID: "f6cf37cf-d28f-47fe-8a27-6585ab6e1579"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:05:09.151168 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.151151    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f6cf37cf-d28f-47fe-8a27-6585ab6e1579" (UID: "f6cf37cf-d28f-47fe-8a27-6585ab6e1579"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:05:09.151285 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.151169    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f6cf37cf-d28f-47fe-8a27-6585ab6e1579" (UID: "f6cf37cf-d28f-47fe-8a27-6585ab6e1579"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:05:09.157045 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.156997    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f6cf37cf-d28f-47fe-8a27-6585ab6e1579" (UID: "f6cf37cf-d28f-47fe-8a27-6585ab6e1579"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:05:09.227250 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.227221    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:09.249545 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.249520    2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-image-registry-private-configuration\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:05:09.249545 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.249540    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-registry-tls\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:05:09.249545 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.249550    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-trusted-ca\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:05:09.249736 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.249559    2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-ca-trust-extracted\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:05:09.249736 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.249567    2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-installation-pull-secrets\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:05:09.249736 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.249576    2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-bound-sa-token\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:05:09.249736 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.249583    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgcfg\" (UniqueName: \"kubernetes.io/projected/f6cf37cf-d28f-47fe-8a27-6585ab6e1579-kube-api-access-vgcfg\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:05:09.324012 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.323978    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:09.424418 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.424344    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:09.716203 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.716127    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:09.914788 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.914752    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:09.961939 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.961908    2571 generic.go:356] "Generic (PLEG): container finished" podID="f6cf37cf-d28f-47fe-8a27-6585ab6e1579" containerID="4055e138e5ac5196beed1fbacaae9607c261d5cc5103283dd965b3ee151f15d1" exitCode=0
Jul 27 21:05:09.962110 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.961967    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s"
Jul 27 21:05:09.962110 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.961948    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s" event={"ID":"f6cf37cf-d28f-47fe-8a27-6585ab6e1579","Type":"ContainerDied","Data":"4055e138e5ac5196beed1fbacaae9607c261d5cc5103283dd965b3ee151f15d1"}
Jul 27 21:05:09.962110 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.962022    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58ff8dd5bb-fv56s" event={"ID":"f6cf37cf-d28f-47fe-8a27-6585ab6e1579","Type":"ContainerDied","Data":"7d6dcc9d02e8a6d1af989a0c3bb154394c129ad39b30dd0b9416725df04135e9"}
Jul 27 21:05:09.962110 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.962050    2571 scope.go:117] "RemoveContainer" containerID="4055e138e5ac5196beed1fbacaae9607c261d5cc5103283dd965b3ee151f15d1"
Jul 27 21:05:09.970725 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.970708    2571 scope.go:117] "RemoveContainer" containerID="4055e138e5ac5196beed1fbacaae9607c261d5cc5103283dd965b3ee151f15d1"
Jul 27 21:05:09.971007 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:05:09.970983    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4055e138e5ac5196beed1fbacaae9607c261d5cc5103283dd965b3ee151f15d1\": container with ID starting with 4055e138e5ac5196beed1fbacaae9607c261d5cc5103283dd965b3ee151f15d1 not found: ID does not exist" containerID="4055e138e5ac5196beed1fbacaae9607c261d5cc5103283dd965b3ee151f15d1"
Jul 27 21:05:09.971087 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.971015    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4055e138e5ac5196beed1fbacaae9607c261d5cc5103283dd965b3ee151f15d1"} err="failed to get container status \"4055e138e5ac5196beed1fbacaae9607c261d5cc5103283dd965b3ee151f15d1\": rpc error: code = NotFound desc = could not find container \"4055e138e5ac5196beed1fbacaae9607c261d5cc5103283dd965b3ee151f15d1\": container with ID starting with 4055e138e5ac5196beed1fbacaae9607c261d5cc5103283dd965b3ee151f15d1 not found: ID does not exist"
Jul 27 21:05:09.985700 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.985677    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58ff8dd5bb-fv56s"]
Jul 27 21:05:09.992769 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.992741    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-58ff8dd5bb-fv56s"]
Jul 27 21:05:09.993220 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:09.993195    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:10.118443 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:10.118414    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:10.220053 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:10.219938    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:10.347574 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:10.347532    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:10.574520 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:10.574486    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6cf37cf-d28f-47fe-8a27-6585ab6e1579" path="/var/lib/kubelet/pods/f6cf37cf-d28f-47fe-8a27-6585ab6e1579/volumes"
Jul 27 21:05:10.899402 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:10.899311    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:11.002969 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:11.002921    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:11.113551 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:11.113520    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:11.210670 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:11.210582    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:11.964683 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:11.964634    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:12.075854 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:12.075829    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:12.163986 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:12.163938    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:15.747443 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:15.747408    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:15.844950 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:15.844919    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:16.000115 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:16.000029    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:16.063586 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:16.063558    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:16.165074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:16.165040    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:16.264880 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:16.264785    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:17.733292 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:17.733241    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:17.799829 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:17.799789    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:22.346931 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:22.346890    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:22.407205 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:22.407172    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:25.535085 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:25.535053    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:25.593377 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:25.593336    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:25.867210 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:25.867125    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:25.930358 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:25.930328    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:29.918459 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:29.918427    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m2bn8"
Jul 27 21:05:32.180212 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:32.180172    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:32.283970 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:32.283930    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:36.523581 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:36.523522    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:40.887031 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:40.886990    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:40.962359 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:40.962329    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:41.191452 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:41.191375    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:41.274321 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:41.274276    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:46.761610 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:46.761580    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xpx7g_5d872ddb-8292-4ecc-b10b-98e6c884724e/serve-healthcheck-canary/0.log"
Jul 27 21:05:50.512808 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:50.512770    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:50.660272 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:50.660240    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:50.738993 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:05:50.738931    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:21.169114 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:06:21.169079    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:21.512712 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:06:21.512627    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:31.022378 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:06:31.022345    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:41.438761 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:06:41.438680    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:41.539415 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:06:41.539376    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:41.593318 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:06:41.593285    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:16.875898 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:07:16.875864    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:16.934477 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:07:16.934438    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:36.442165 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:07:36.442115    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:36.498334 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:07:36.498294    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:46.728788 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:07:46.728736    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:46.798223 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:07:46.798180    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:48.397625 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:07:48.397587    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:48.461488 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:07:48.461455    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:55.882075 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:07:55.882037    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:55.982512 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:07:55.982486    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:57.719946 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:07:57.719912    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:57.820232 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:07:57.820189    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:57.879328 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:07:57.879288    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:08:40.416212 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:08:40.416188    2571 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 21:09:05.813739 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:09:05.813695    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:35.533378 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:09:35.533286    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:35.589006 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:09:35.588975    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:35.942465 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:09:35.942384    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:36.045093 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:09:36.045062    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:36.154720 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:09:36.154687    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:59.008509 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:09:59.008460    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:59.143476 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:09:59.143441    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:10:05.499661 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:10:05.499626    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:10:05.834847 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:10:05.834806    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:10:06.027585 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:10:06.027538    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:10:06.199638 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:10:06.199556    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:10:06.295247 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:10:06.295204    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:10:06.642888 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:10:06.642852    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:10:06.931870 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:10:06.931783    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:10:07.060685 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:10:07.060647    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:10:07.168548 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:10:07.168515    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:10:07.345278 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:10:07.345248    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:10:07.694637 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:10:07.694553    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:10:07.879003 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:10:07.878969    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:10:07.987730 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:10:07.987645    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:26.376469 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:11:26.376433    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:26.488823 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:11:26.488788    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:26.676762 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:11:26.676664    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:26.821085 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:11:26.821049    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:27.075376 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:11:27.075343    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:27.178552 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:11:27.178518    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:27.275479 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:11:27.275445    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:41.569568 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:11:41.569533    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:07.043857 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:12:07.043810    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:07.144102 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:12:07.144071    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:07.207821 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:12:07.207784    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:07.689629 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:12:07.689592    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:07.791696 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:12:07.791659    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:17.890537 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:12:17.890500    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:17.990498 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:12:17.990458    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:18.087863 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:12:18.087832    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:44.805776 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:12:44.805689    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:44.885257 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:12:44.885220    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:44.960242 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:12:44.960200    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:45.602246 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:12:45.602204    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:45.664458 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:12:45.664430    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:13:10.021807 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:13:10.021749    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:19.324943 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:19.324910    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:21.914499 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:21.914455    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:21.978337 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:21.978288    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:25.943390 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:25.943356    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:34.279252 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.279211    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:34.352401 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.352364    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:34.462519 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.462486    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:34.592341 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.592260    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:34.690098 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.690060    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:34.716204 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.716175    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"]
Jul 27 21:14:34.716429 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.716416    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6cf37cf-d28f-47fe-8a27-6585ab6e1579" containerName="registry"
Jul 27 21:14:34.716491 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.716430    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6cf37cf-d28f-47fe-8a27-6585ab6e1579" containerName="registry"
Jul 27 21:14:34.716491 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.716479    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6cf37cf-d28f-47fe-8a27-6585ab6e1579" containerName="registry"
Jul 27 21:14:34.718543 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.718528    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"
Jul 27 21:14:34.756746 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.756672    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:14:34.756746 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.756711    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-070ff-predictor-serving-cert\""
Jul 27 21:14:34.757034 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.756719    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-070ff-kube-rbac-proxy-sar-config\""
Jul 27 21:14:34.757680 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.757664    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:14:34.757781 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.757663    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wg596\""
Jul 27 21:14:34.850735 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.850657    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"]
Jul 27 21:14:34.866817 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.866777    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vfqv\" (UniqueName: \"kubernetes.io/projected/9a2b84e8-0b58-4151-96e6-467b6b90d0bb-kube-api-access-4vfqv\") pod \"error-404-isvc-070ff-predictor-59657fff56-glz8v\" (UID: \"9a2b84e8-0b58-4151-96e6-467b6b90d0bb\") " pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"
Jul 27 21:14:34.867005 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.866850    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a2b84e8-0b58-4151-96e6-467b6b90d0bb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-070ff-predictor-59657fff56-glz8v\" (UID: \"9a2b84e8-0b58-4151-96e6-467b6b90d0bb\") " pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"
Jul 27 21:14:34.867005 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.866990    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a2b84e8-0b58-4151-96e6-467b6b90d0bb-proxy-tls\") pod \"error-404-isvc-070ff-predictor-59657fff56-glz8v\" (UID: \"9a2b84e8-0b58-4151-96e6-467b6b90d0bb\") " pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"
Jul 27 21:14:34.957918 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.957880    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:34.967713 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.967669    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a2b84e8-0b58-4151-96e6-467b6b90d0bb-proxy-tls\") pod \"error-404-isvc-070ff-predictor-59657fff56-glz8v\" (UID: \"9a2b84e8-0b58-4151-96e6-467b6b90d0bb\") " pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"
Jul 27 21:14:34.967886 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.967730    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vfqv\" (UniqueName: \"kubernetes.io/projected/9a2b84e8-0b58-4151-96e6-467b6b90d0bb-kube-api-access-4vfqv\") pod \"error-404-isvc-070ff-predictor-59657fff56-glz8v\" (UID: \"9a2b84e8-0b58-4151-96e6-467b6b90d0bb\") " pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"
Jul 27 21:14:34.967886 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.967762    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a2b84e8-0b58-4151-96e6-467b6b90d0bb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-070ff-predictor-59657fff56-glz8v\" (UID: \"9a2b84e8-0b58-4151-96e6-467b6b90d0bb\") " pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"
Jul 27 21:14:34.968480 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.968455    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a2b84e8-0b58-4151-96e6-467b6b90d0bb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-070ff-predictor-59657fff56-glz8v\" (UID: \"9a2b84e8-0b58-4151-96e6-467b6b90d0bb\") " pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"
Jul 27 21:14:34.970617 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.970595    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a2b84e8-0b58-4151-96e6-467b6b90d0bb-proxy-tls\") pod \"error-404-isvc-070ff-predictor-59657fff56-glz8v\" (UID: \"9a2b84e8-0b58-4151-96e6-467b6b90d0bb\") " pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"
Jul 27 21:14:34.977072 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:34.977049    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vfqv\" (UniqueName: \"kubernetes.io/projected/9a2b84e8-0b58-4151-96e6-467b6b90d0bb-kube-api-access-4vfqv\") pod \"error-404-isvc-070ff-predictor-59657fff56-glz8v\" (UID: \"9a2b84e8-0b58-4151-96e6-467b6b90d0bb\") " pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"
Jul 27 21:14:35.030740 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:35.030701    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"
Jul 27 21:14:35.049115 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:35.049040    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:35.181476 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:35.181454    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"]
Jul 27 21:14:35.184367 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:14:35.184337    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2b84e8_0b58_4151_96e6_467b6b90d0bb.slice/crio-8320b3ed6aaaf511a7d849925a099189aeaf6a92dffc89997e2a3d6459e5e551 WatchSource:0}: Error finding container 8320b3ed6aaaf511a7d849925a099189aeaf6a92dffc89997e2a3d6459e5e551: Status 404 returned error can't find the container with id 8320b3ed6aaaf511a7d849925a099189aeaf6a92dffc89997e2a3d6459e5e551
Jul 27 21:14:35.186746 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:35.186726    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:14:35.278138 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:35.278106    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:35.378860 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:35.378778    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:35.499101 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:35.499046    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:35.824758 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:35.824715    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v" event={"ID":"9a2b84e8-0b58-4151-96e6-467b6b90d0bb","Type":"ContainerStarted","Data":"8320b3ed6aaaf511a7d849925a099189aeaf6a92dffc89997e2a3d6459e5e551"}
Jul 27 21:14:47.878378 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:47.878335    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v" event={"ID":"9a2b84e8-0b58-4151-96e6-467b6b90d0bb","Type":"ContainerStarted","Data":"b4007954d045adaf3afbe5dc4a9cb1a4e3d669b43c9fd6f5d992aceec392d468"}
Jul 27 21:14:49.886723 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:49.886686    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v" event={"ID":"9a2b84e8-0b58-4151-96e6-467b6b90d0bb","Type":"ContainerStarted","Data":"0e64c6706969f4ede7bc542a32f8fbbde05a87dbbba29dbd8206fd3a3f167f1b"}
Jul 27 21:14:49.887216 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:49.886748    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"
Jul 27 21:14:49.887216 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:49.886813    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"
Jul 27 21:14:49.888346 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:49.888318    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v" podUID="9a2b84e8-0b58-4151-96e6-467b6b90d0bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 27 21:14:49.908223 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:49.908176    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v" podStartSLOduration=1.7678241369999999 podStartE2EDuration="15.908159994s" totalImagesPullingTime="14.140335857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:14:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:14:49.906831879 +0000 UTC m=+669.842048271" watchObservedRunningTime="2026-07-27 21:14:49.908159994 +0000 UTC m=+669.843376372"
Jul 27 21:14:50.891075 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:50.891033    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v" podUID="9a2b84e8-0b58-4151-96e6-467b6b90d0bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 27 21:14:54.424932 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:54.424884    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:55.896193 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:55.896163    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"
Jul 27 21:14:55.896574 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:14:55.896545    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v" podUID="9a2b84e8-0b58-4151-96e6-467b6b90d0bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 27 21:15:05.896704 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:15:05.896666    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v" podUID="9a2b84e8-0b58-4151-96e6-467b6b90d0bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 27 21:15:15.896881 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:15:15.896834    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v" podUID="9a2b84e8-0b58-4151-96e6-467b6b90d0bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 27 21:15:25.897536 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:15:25.897494    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v" podUID="9a2b84e8-0b58-4151-96e6-467b6b90d0bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 27 21:15:35.897209 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:15:35.897129    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"
Jul 27 21:16:14.249897 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:14.249867    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596"]
Jul 27 21:16:14.253075 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:14.253060    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596"
Jul 27 21:16:14.255237 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:14.255219    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-070ff-kube-rbac-proxy-sar-config\""
Jul 27 21:16:14.255341 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:14.255305    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-070ff-serving-cert\""
Jul 27 21:16:14.261443 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:14.261244    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:14.262503 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:14.262474    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596"]
Jul 27 21:16:14.411265 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:14.411229    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b0c3c35-819a-45b8-beab-4cb0fffeafe6-proxy-tls\") pod \"switch-graph-070ff-5949b59c9c-h4596\" (UID: \"0b0c3c35-819a-45b8-beab-4cb0fffeafe6\") " pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596"
Jul 27 21:16:14.411440 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:14.411289    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0c3c35-819a-45b8-beab-4cb0fffeafe6-openshift-service-ca-bundle\") pod \"switch-graph-070ff-5949b59c9c-h4596\" (UID: \"0b0c3c35-819a-45b8-beab-4cb0fffeafe6\") " pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596"
Jul 27 21:16:14.512045 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:14.511936    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b0c3c35-819a-45b8-beab-4cb0fffeafe6-proxy-tls\") pod \"switch-graph-070ff-5949b59c9c-h4596\" (UID: \"0b0c3c35-819a-45b8-beab-4cb0fffeafe6\") " pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596"
Jul 27 21:16:14.512045 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:14.512036    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0c3c35-819a-45b8-beab-4cb0fffeafe6-openshift-service-ca-bundle\") pod \"switch-graph-070ff-5949b59c9c-h4596\" (UID: \"0b0c3c35-819a-45b8-beab-4cb0fffeafe6\") " pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596"
Jul 27 21:16:14.512687 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:14.512663    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0c3c35-819a-45b8-beab-4cb0fffeafe6-openshift-service-ca-bundle\") pod \"switch-graph-070ff-5949b59c9c-h4596\" (UID: \"0b0c3c35-819a-45b8-beab-4cb0fffeafe6\") " pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596"
Jul 27 21:16:14.514574 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:14.514555    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b0c3c35-819a-45b8-beab-4cb0fffeafe6-proxy-tls\") pod \"switch-graph-070ff-5949b59c9c-h4596\" (UID: \"0b0c3c35-819a-45b8-beab-4cb0fffeafe6\") " pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596"
Jul 27 21:16:14.549664 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:14.549634    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:14.564462 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:14.564440    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596"
Jul 27 21:16:14.687532 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:14.687511    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596"]
Jul 27 21:16:14.689872 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:16:14.689830    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0c3c35_819a_45b8_beab_4cb0fffeafe6.slice/crio-df5256dc513e005d5376b03e52680d1870aacc7aa16b13d9eed13570942df2f9 WatchSource:0}: Error finding container df5256dc513e005d5376b03e52680d1870aacc7aa16b13d9eed13570942df2f9: Status 404 returned error can't find the container with id df5256dc513e005d5376b03e52680d1870aacc7aa16b13d9eed13570942df2f9
Jul 27 21:16:15.166638 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:15.166602    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596" event={"ID":"0b0c3c35-819a-45b8-beab-4cb0fffeafe6","Type":"ContainerStarted","Data":"df5256dc513e005d5376b03e52680d1870aacc7aa16b13d9eed13570942df2f9"}
Jul 27 21:16:18.178659 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:18.178629    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596" event={"ID":"0b0c3c35-819a-45b8-beab-4cb0fffeafe6","Type":"ContainerStarted","Data":"05b72928edda8250aaff96a4d04f9909eeb8b240c42e6aad0ed1ac1b8360b416"}
Jul 27 21:16:18.179081 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:18.178689    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596"
Jul 27 21:16:18.195152 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:18.195106    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596" podStartSLOduration=1.682926234 podStartE2EDuration="4.195091762s" totalImagesPullingTime="2.512165528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:16:18.193639222 +0000 UTC m=+758.128855600" watchObservedRunningTime="2026-07-27 21:16:18.195091762 +0000 UTC m=+758.130308150"
Jul 27 21:16:24.187579 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.187553    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596"
Jul 27 21:16:24.421080 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.421043    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596"]
Jul 27 21:16:24.421344 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.421296    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596" podUID="0b0c3c35-819a-45b8-beab-4cb0fffeafe6" containerName="switch-graph-070ff" containerID="cri-o://05b72928edda8250aaff96a4d04f9909eeb8b240c42e6aad0ed1ac1b8360b416" gracePeriod=30
Jul 27 21:16:24.446024 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.445937    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:24.598876 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.598840    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:24.699600 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.699508    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:24.710513 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.710488    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"]
Jul 27 21:16:24.713652 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.713637    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"
Jul 27 21:16:24.716010 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.715993    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d355e-kube-rbac-proxy-sar-config\""
Jul 27 21:16:24.716108 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.715992    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d355e-predictor-serving-cert\""
Jul 27 21:16:24.724116 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.724094    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"]
Jul 27 21:16:24.760231 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.760202    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"]
Jul 27 21:16:24.760496 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.760471    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v" podUID="9a2b84e8-0b58-4151-96e6-467b6b90d0bb" containerName="kserve-container" containerID="cri-o://b4007954d045adaf3afbe5dc4a9cb1a4e3d669b43c9fd6f5d992aceec392d468" gracePeriod=30
Jul 27 21:16:24.760571 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.760502    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v" podUID="9a2b84e8-0b58-4151-96e6-467b6b90d0bb" containerName="kube-rbac-proxy" containerID="cri-o://0e64c6706969f4ede7bc542a32f8fbbde05a87dbbba29dbd8206fd3a3f167f1b" gracePeriod=30
Jul 27 21:16:24.770682 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.770650    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:24.789950 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.789927    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ca8209c-921e-48e2-9de4-82a1d5d410b1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d355e-predictor-77499898bb-cghz4\" (UID: \"1ca8209c-921e-48e2-9de4-82a1d5d410b1\") " pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"
Jul 27 21:16:24.790095 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.789975    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sm9b\" (UniqueName: \"kubernetes.io/projected/1ca8209c-921e-48e2-9de4-82a1d5d410b1-kube-api-access-5sm9b\") pod \"error-404-isvc-d355e-predictor-77499898bb-cghz4\" (UID: \"1ca8209c-921e-48e2-9de4-82a1d5d410b1\") " pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"
Jul 27 21:16:24.790095 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.790019    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ca8209c-921e-48e2-9de4-82a1d5d410b1-proxy-tls\") pod \"error-404-isvc-d355e-predictor-77499898bb-cghz4\" (UID: \"1ca8209c-921e-48e2-9de4-82a1d5d410b1\") " pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"
Jul 27 21:16:24.837725 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.837689    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:24.890577 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.890548    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ca8209c-921e-48e2-9de4-82a1d5d410b1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d355e-predictor-77499898bb-cghz4\" (UID: \"1ca8209c-921e-48e2-9de4-82a1d5d410b1\") " pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"
Jul 27 21:16:24.890722 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.890596    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sm9b\" (UniqueName: \"kubernetes.io/projected/1ca8209c-921e-48e2-9de4-82a1d5d410b1-kube-api-access-5sm9b\") pod \"error-404-isvc-d355e-predictor-77499898bb-cghz4\" (UID: \"1ca8209c-921e-48e2-9de4-82a1d5d410b1\") " pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"
Jul 27 21:16:24.890722 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.890650    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ca8209c-921e-48e2-9de4-82a1d5d410b1-proxy-tls\") pod \"error-404-isvc-d355e-predictor-77499898bb-cghz4\" (UID: \"1ca8209c-921e-48e2-9de4-82a1d5d410b1\") " pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"
Jul 27 21:16:24.890836 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:16:24.890767    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-d355e-predictor-serving-cert: secret "error-404-isvc-d355e-predictor-serving-cert" not found
Jul 27 21:16:24.890885 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:16:24.890839    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ca8209c-921e-48e2-9de4-82a1d5d410b1-proxy-tls podName:1ca8209c-921e-48e2-9de4-82a1d5d410b1 nodeName:}" failed. No retries permitted until 2026-07-27 21:16:25.390815329 +0000 UTC m=+765.326031690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1ca8209c-921e-48e2-9de4-82a1d5d410b1-proxy-tls") pod "error-404-isvc-d355e-predictor-77499898bb-cghz4" (UID: "1ca8209c-921e-48e2-9de4-82a1d5d410b1") : secret "error-404-isvc-d355e-predictor-serving-cert" not found
Jul 27 21:16:24.891390 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.891372    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ca8209c-921e-48e2-9de4-82a1d5d410b1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d355e-predictor-77499898bb-cghz4\" (UID: \"1ca8209c-921e-48e2-9de4-82a1d5d410b1\") " pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"
Jul 27 21:16:24.899062 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:24.899043    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sm9b\" (UniqueName: \"kubernetes.io/projected/1ca8209c-921e-48e2-9de4-82a1d5d410b1-kube-api-access-5sm9b\") pod \"error-404-isvc-d355e-predictor-77499898bb-cghz4\" (UID: \"1ca8209c-921e-48e2-9de4-82a1d5d410b1\") " pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"
Jul 27 21:16:25.024203 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:25.024169    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:25.125179 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:25.125148    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:25.203277 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:25.203245    2571 generic.go:356] "Generic (PLEG): container finished" podID="9a2b84e8-0b58-4151-96e6-467b6b90d0bb" containerID="0e64c6706969f4ede7bc542a32f8fbbde05a87dbbba29dbd8206fd3a3f167f1b" exitCode=2
Jul 27 21:16:25.203642 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:25.203317    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v" event={"ID":"9a2b84e8-0b58-4151-96e6-467b6b90d0bb","Type":"ContainerDied","Data":"0e64c6706969f4ede7bc542a32f8fbbde05a87dbbba29dbd8206fd3a3f167f1b"}
Jul 27 21:16:25.393094 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:25.393018    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ca8209c-921e-48e2-9de4-82a1d5d410b1-proxy-tls\") pod \"error-404-isvc-d355e-predictor-77499898bb-cghz4\" (UID: \"1ca8209c-921e-48e2-9de4-82a1d5d410b1\") " pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"
Jul 27 21:16:25.395599 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:25.395576    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ca8209c-921e-48e2-9de4-82a1d5d410b1-proxy-tls\") pod \"error-404-isvc-d355e-predictor-77499898bb-cghz4\" (UID: \"1ca8209c-921e-48e2-9de4-82a1d5d410b1\") " pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"
Jul 27 21:16:25.623906 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:25.623864    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"
Jul 27 21:16:25.750705 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:25.750553    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"]
Jul 27 21:16:25.753252 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:16:25.753220    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca8209c_921e_48e2_9de4_82a1d5d410b1.slice/crio-86997674c4adfe7bc8b5c3c2afc58aa538193f60132992688e5a4444620bc2b5 WatchSource:0}: Error finding container 86997674c4adfe7bc8b5c3c2afc58aa538193f60132992688e5a4444620bc2b5: Status 404 returned error can't find the container with id 86997674c4adfe7bc8b5c3c2afc58aa538193f60132992688e5a4444620bc2b5
Jul 27 21:16:25.891876 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:25.891838    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v" podUID="9a2b84e8-0b58-4151-96e6-467b6b90d0bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.16:8643/healthz\": dial tcp 10.133.0.16:8643: connect: connection refused"
Jul 27 21:16:25.897321 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:25.897283    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v" podUID="9a2b84e8-0b58-4151-96e6-467b6b90d0bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 27 21:16:26.207972 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:26.207911    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4" event={"ID":"1ca8209c-921e-48e2-9de4-82a1d5d410b1","Type":"ContainerStarted","Data":"a5550f913154f188c37a67d9c4029f12841c195ab3bed382ea65bc8292c171de"}
Jul 27 21:16:26.208379 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:26.208002    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4" event={"ID":"1ca8209c-921e-48e2-9de4-82a1d5d410b1","Type":"ContainerStarted","Data":"c671a7e5838881205d5f80a3c2965ef15d8b400a670a00100239de3d8364f22a"}
Jul 27 21:16:26.208379 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:26.208037    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4" event={"ID":"1ca8209c-921e-48e2-9de4-82a1d5d410b1","Type":"ContainerStarted","Data":"86997674c4adfe7bc8b5c3c2afc58aa538193f60132992688e5a4444620bc2b5"}
Jul 27 21:16:26.208379 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:26.208068    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"
Jul 27 21:16:26.208379 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:26.208096    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"
Jul 27 21:16:26.209517 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:26.209490    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4" podUID="1ca8209c-921e-48e2-9de4-82a1d5d410b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 27 21:16:26.224914 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:26.224870    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4" podStartSLOduration=2.224855408 podStartE2EDuration="2.224855408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:16:26.224246128 +0000 UTC m=+766.159462509" watchObservedRunningTime="2026-07-27 21:16:26.224855408 +0000 UTC m=+766.160071787"
Jul 27 21:16:27.211908 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:27.211868    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4" podUID="1ca8209c-921e-48e2-9de4-82a1d5d410b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 27 21:16:27.912890 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:27.912866    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"
Jul 27 21:16:28.009317 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:28.009227    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vfqv\" (UniqueName: \"kubernetes.io/projected/9a2b84e8-0b58-4151-96e6-467b6b90d0bb-kube-api-access-4vfqv\") pod \"9a2b84e8-0b58-4151-96e6-467b6b90d0bb\" (UID: \"9a2b84e8-0b58-4151-96e6-467b6b90d0bb\") "
Jul 27 21:16:28.009317 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:28.009276    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a2b84e8-0b58-4151-96e6-467b6b90d0bb-kube-rbac-proxy-sar-config\") pod \"9a2b84e8-0b58-4151-96e6-467b6b90d0bb\" (UID: \"9a2b84e8-0b58-4151-96e6-467b6b90d0bb\") "
Jul 27 21:16:28.009317 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:28.009310    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a2b84e8-0b58-4151-96e6-467b6b90d0bb-proxy-tls\") pod \"9a2b84e8-0b58-4151-96e6-467b6b90d0bb\" (UID: \"9a2b84e8-0b58-4151-96e6-467b6b90d0bb\") "
Jul 27 21:16:28.009640 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:28.009613    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a2b84e8-0b58-4151-96e6-467b6b90d0bb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9a2b84e8-0b58-4151-96e6-467b6b90d0bb" (UID: "9a2b84e8-0b58-4151-96e6-467b6b90d0bb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:16:28.011703 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:28.011667    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a2b84e8-0b58-4151-96e6-467b6b90d0bb-kube-api-access-4vfqv" (OuterVolumeSpecName: "kube-api-access-4vfqv") pod "9a2b84e8-0b58-4151-96e6-467b6b90d0bb" (UID: "9a2b84e8-0b58-4151-96e6-467b6b90d0bb"). InnerVolumeSpecName "kube-api-access-4vfqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:16:28.011703 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:28.011696    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a2b84e8-0b58-4151-96e6-467b6b90d0bb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9a2b84e8-0b58-4151-96e6-467b6b90d0bb" (UID: "9a2b84e8-0b58-4151-96e6-467b6b90d0bb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:16:28.110050 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:28.110021    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a2b84e8-0b58-4151-96e6-467b6b90d0bb-proxy-tls\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:16:28.110050 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:28.110045    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vfqv\" (UniqueName: \"kubernetes.io/projected/9a2b84e8-0b58-4151-96e6-467b6b90d0bb-kube-api-access-4vfqv\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:16:28.110207 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:28.110058    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a2b84e8-0b58-4151-96e6-467b6b90d0bb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:16:28.221202 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:28.221168    2571 generic.go:356] "Generic (PLEG): container finished" podID="9a2b84e8-0b58-4151-96e6-467b6b90d0bb" containerID="b4007954d045adaf3afbe5dc4a9cb1a4e3d669b43c9fd6f5d992aceec392d468" exitCode=0
Jul 27 21:16:28.221557 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:28.221224    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"
Jul 27 21:16:28.221557 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:28.221252    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v" event={"ID":"9a2b84e8-0b58-4151-96e6-467b6b90d0bb","Type":"ContainerDied","Data":"b4007954d045adaf3afbe5dc4a9cb1a4e3d669b43c9fd6f5d992aceec392d468"}
Jul 27 21:16:28.221557 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:28.221328    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v" event={"ID":"9a2b84e8-0b58-4151-96e6-467b6b90d0bb","Type":"ContainerDied","Data":"8320b3ed6aaaf511a7d849925a099189aeaf6a92dffc89997e2a3d6459e5e551"}
Jul 27 21:16:28.221557 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:28.221353    2571 scope.go:117] "RemoveContainer" containerID="0e64c6706969f4ede7bc542a32f8fbbde05a87dbbba29dbd8206fd3a3f167f1b"
Jul 27 21:16:28.229451 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:28.229435    2571 scope.go:117] "RemoveContainer" containerID="b4007954d045adaf3afbe5dc4a9cb1a4e3d669b43c9fd6f5d992aceec392d468"
Jul 27 21:16:28.236271 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:28.236250    2571 scope.go:117] "RemoveContainer" containerID="0e64c6706969f4ede7bc542a32f8fbbde05a87dbbba29dbd8206fd3a3f167f1b"
Jul 27 21:16:28.236495 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:16:28.236476    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e64c6706969f4ede7bc542a32f8fbbde05a87dbbba29dbd8206fd3a3f167f1b\": container with ID starting with 0e64c6706969f4ede7bc542a32f8fbbde05a87dbbba29dbd8206fd3a3f167f1b not found: ID does not exist" containerID="0e64c6706969f4ede7bc542a32f8fbbde05a87dbbba29dbd8206fd3a3f167f1b"
Jul 27 21:16:28.236557 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:28.236503    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e64c6706969f4ede7bc542a32f8fbbde05a87dbbba29dbd8206fd3a3f167f1b"} err="failed to get container status \"0e64c6706969f4ede7bc542a32f8fbbde05a87dbbba29dbd8206fd3a3f167f1b\": rpc error: code = NotFound desc = could not find container \"0e64c6706969f4ede7bc542a32f8fbbde05a87dbbba29dbd8206fd3a3f167f1b\": container with ID starting with 0e64c6706969f4ede7bc542a32f8fbbde05a87dbbba29dbd8206fd3a3f167f1b not found: ID does not exist"
Jul 27 21:16:28.236557 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:28.236521    2571 scope.go:117] "RemoveContainer" containerID="b4007954d045adaf3afbe5dc4a9cb1a4e3d669b43c9fd6f5d992aceec392d468"
Jul 27 21:16:28.236707 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:16:28.236693    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4007954d045adaf3afbe5dc4a9cb1a4e3d669b43c9fd6f5d992aceec392d468\": container with ID starting with b4007954d045adaf3afbe5dc4a9cb1a4e3d669b43c9fd6f5d992aceec392d468 not found: ID does not exist" containerID="b4007954d045adaf3afbe5dc4a9cb1a4e3d669b43c9fd6f5d992aceec392d468"
Jul 27 21:16:28.236746 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:28.236710    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4007954d045adaf3afbe5dc4a9cb1a4e3d669b43c9fd6f5d992aceec392d468"} err="failed to get container status \"b4007954d045adaf3afbe5dc4a9cb1a4e3d669b43c9fd6f5d992aceec392d468\": rpc error: code = NotFound desc = could not find container \"b4007954d045adaf3afbe5dc4a9cb1a4e3d669b43c9fd6f5d992aceec392d468\": container with ID starting with b4007954d045adaf3afbe5dc4a9cb1a4e3d669b43c9fd6f5d992aceec392d468 not found: ID does not exist"
Jul 27 21:16:28.240698 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:28.240677    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"]
Jul 27 21:16:28.245242 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:28.245224    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-070ff-predictor-59657fff56-glz8v"]
Jul 27 21:16:28.571920 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:28.571887    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a2b84e8-0b58-4151-96e6-467b6b90d0bb" path="/var/lib/kubelet/pods/9a2b84e8-0b58-4151-96e6-467b6b90d0bb/volumes"
Jul 27 21:16:29.186373 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:29.186338    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596" podUID="0b0c3c35-819a-45b8-beab-4cb0fffeafe6" containerName="switch-graph-070ff" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:16:32.216318 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:32.216290    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"
Jul 27 21:16:32.216820 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:32.216797    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4" podUID="1ca8209c-921e-48e2-9de4-82a1d5d410b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 27 21:16:34.186850 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:34.186814    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596" podUID="0b0c3c35-819a-45b8-beab-4cb0fffeafe6" containerName="switch-graph-070ff" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:16:39.186491 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:39.186453    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596" podUID="0b0c3c35-819a-45b8-beab-4cb0fffeafe6" containerName="switch-graph-070ff" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:16:39.186927 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:39.186562    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596"
Jul 27 21:16:42.217191 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:42.217151    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4" podUID="1ca8209c-921e-48e2-9de4-82a1d5d410b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 27 21:16:44.186594 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:44.186557    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596" podUID="0b0c3c35-819a-45b8-beab-4cb0fffeafe6" containerName="switch-graph-070ff" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:16:44.248375 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:44.248340    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:44.349228 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:44.349191    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:49.186673 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:49.186624    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596" podUID="0b0c3c35-819a-45b8-beab-4cb0fffeafe6" containerName="switch-graph-070ff" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:16:52.217551 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:52.217517    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4" podUID="1ca8209c-921e-48e2-9de4-82a1d5d410b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 27 21:16:54.186628 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.186586    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596" podUID="0b0c3c35-819a-45b8-beab-4cb0fffeafe6" containerName="switch-graph-070ff" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:16:54.362195 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.362157    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:54.482149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.482113    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:54.587051 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.586832    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:54.603391 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.603367    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596"
Jul 27 21:16:54.677488 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.677459    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"]
Jul 27 21:16:54.677726 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.677714    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a2b84e8-0b58-4151-96e6-467b6b90d0bb" containerName="kserve-container"
Jul 27 21:16:54.677816 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.677731    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a2b84e8-0b58-4151-96e6-467b6b90d0bb" containerName="kserve-container"
Jul 27 21:16:54.677816 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.677744    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a2b84e8-0b58-4151-96e6-467b6b90d0bb" containerName="kube-rbac-proxy"
Jul 27 21:16:54.677816 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.677750    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a2b84e8-0b58-4151-96e6-467b6b90d0bb" containerName="kube-rbac-proxy"
Jul 27 21:16:54.677816 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.677757    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0c3c35-819a-45b8-beab-4cb0fffeafe6" containerName="switch-graph-070ff"
Jul 27 21:16:54.677816 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.677762    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0c3c35-819a-45b8-beab-4cb0fffeafe6" containerName="switch-graph-070ff"
Jul 27 21:16:54.677992 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.677816    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a2b84e8-0b58-4151-96e6-467b6b90d0bb" containerName="kserve-container"
Jul 27 21:16:54.677992 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.677826    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0c3c35-819a-45b8-beab-4cb0fffeafe6" containerName="switch-graph-070ff"
Jul 27 21:16:54.677992 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.677832    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a2b84e8-0b58-4151-96e6-467b6b90d0bb" containerName="kube-rbac-proxy"
Jul 27 21:16:54.680746 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.680726    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"
Jul 27 21:16:54.682943 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.682918    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d13b2-predictor-serving-cert\""
Jul 27 21:16:54.683086 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.682994    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d13b2-kube-rbac-proxy-sar-config\""
Jul 27 21:16:54.684940 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.684916    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:54.691234 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.691215    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"]
Jul 27 21:16:54.699935 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.699916    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b0c3c35-819a-45b8-beab-4cb0fffeafe6-proxy-tls\") pod \"0b0c3c35-819a-45b8-beab-4cb0fffeafe6\" (UID: \"0b0c3c35-819a-45b8-beab-4cb0fffeafe6\") "
Jul 27 21:16:54.700051 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.699987    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0c3c35-819a-45b8-beab-4cb0fffeafe6-openshift-service-ca-bundle\") pod \"0b0c3c35-819a-45b8-beab-4cb0fffeafe6\" (UID: \"0b0c3c35-819a-45b8-beab-4cb0fffeafe6\") "
Jul 27 21:16:54.700314 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.700296    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0c3c35-819a-45b8-beab-4cb0fffeafe6-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0b0c3c35-819a-45b8-beab-4cb0fffeafe6" (UID: "0b0c3c35-819a-45b8-beab-4cb0fffeafe6"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:16:54.702238 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.702218    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0c3c35-819a-45b8-beab-4cb0fffeafe6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0b0c3c35-819a-45b8-beab-4cb0fffeafe6" (UID: "0b0c3c35-819a-45b8-beab-4cb0fffeafe6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:16:54.780515 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.780483    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:54.800713 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.800683    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/840489b6-c02a-4575-903d-99faaf200107-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d13b2-predictor-557774b68b-xp7n6\" (UID: \"840489b6-c02a-4575-903d-99faaf200107\") " pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"
Jul 27 21:16:54.800883 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.800739    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vff7v\" (UniqueName: \"kubernetes.io/projected/840489b6-c02a-4575-903d-99faaf200107-kube-api-access-vff7v\") pod \"error-404-isvc-d13b2-predictor-557774b68b-xp7n6\" (UID: \"840489b6-c02a-4575-903d-99faaf200107\") " pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"
Jul 27 21:16:54.800883 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.800830    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/840489b6-c02a-4575-903d-99faaf200107-proxy-tls\") pod \"error-404-isvc-d13b2-predictor-557774b68b-xp7n6\" (UID: \"840489b6-c02a-4575-903d-99faaf200107\") " pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"
Jul 27 21:16:54.801006 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.800904    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b0c3c35-819a-45b8-beab-4cb0fffeafe6-proxy-tls\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:16:54.801006 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.800922    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0c3c35-819a-45b8-beab-4cb0fffeafe6-openshift-service-ca-bundle\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:16:54.901411 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.901378    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/840489b6-c02a-4575-903d-99faaf200107-proxy-tls\") pod \"error-404-isvc-d13b2-predictor-557774b68b-xp7n6\" (UID: \"840489b6-c02a-4575-903d-99faaf200107\") " pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"
Jul 27 21:16:54.901594 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.901435    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/840489b6-c02a-4575-903d-99faaf200107-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d13b2-predictor-557774b68b-xp7n6\" (UID: \"840489b6-c02a-4575-903d-99faaf200107\") " pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"
Jul 27 21:16:54.901594 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.901478    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vff7v\" (UniqueName: \"kubernetes.io/projected/840489b6-c02a-4575-903d-99faaf200107-kube-api-access-vff7v\") pod \"error-404-isvc-d13b2-predictor-557774b68b-xp7n6\" (UID: \"840489b6-c02a-4575-903d-99faaf200107\") " pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"
Jul 27 21:16:54.901594 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:16:54.901561    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-serving-cert: secret "error-404-isvc-d13b2-predictor-serving-cert" not found
Jul 27 21:16:54.901764 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:16:54.901636    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/840489b6-c02a-4575-903d-99faaf200107-proxy-tls podName:840489b6-c02a-4575-903d-99faaf200107 nodeName:}" failed. No retries permitted until 2026-07-27 21:16:55.401613932 +0000 UTC m=+795.336830293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/840489b6-c02a-4575-903d-99faaf200107-proxy-tls") pod "error-404-isvc-d13b2-predictor-557774b68b-xp7n6" (UID: "840489b6-c02a-4575-903d-99faaf200107") : secret "error-404-isvc-d13b2-predictor-serving-cert" not found
Jul 27 21:16:54.902109 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.902091    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/840489b6-c02a-4575-903d-99faaf200107-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d13b2-predictor-557774b68b-xp7n6\" (UID: \"840489b6-c02a-4575-903d-99faaf200107\") " pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"
Jul 27 21:16:54.910187 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:54.910162    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vff7v\" (UniqueName: \"kubernetes.io/projected/840489b6-c02a-4575-903d-99faaf200107-kube-api-access-vff7v\") pod \"error-404-isvc-d13b2-predictor-557774b68b-xp7n6\" (UID: \"840489b6-c02a-4575-903d-99faaf200107\") " pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"
Jul 27 21:16:55.076800 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:55.076767    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:55.318557 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:55.318527    2571 generic.go:356] "Generic (PLEG): container finished" podID="0b0c3c35-819a-45b8-beab-4cb0fffeafe6" containerID="05b72928edda8250aaff96a4d04f9909eeb8b240c42e6aad0ed1ac1b8360b416" exitCode=0
Jul 27 21:16:55.319005 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:55.318571    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596"
Jul 27 21:16:55.319005 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:55.318586    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596" event={"ID":"0b0c3c35-819a-45b8-beab-4cb0fffeafe6","Type":"ContainerDied","Data":"05b72928edda8250aaff96a4d04f9909eeb8b240c42e6aad0ed1ac1b8360b416"}
Jul 27 21:16:55.319005 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:55.318630    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596" event={"ID":"0b0c3c35-819a-45b8-beab-4cb0fffeafe6","Type":"ContainerDied","Data":"df5256dc513e005d5376b03e52680d1870aacc7aa16b13d9eed13570942df2f9"}
Jul 27 21:16:55.319005 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:55.318651    2571 scope.go:117] "RemoveContainer" containerID="05b72928edda8250aaff96a4d04f9909eeb8b240c42e6aad0ed1ac1b8360b416"
Jul 27 21:16:55.326705 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:55.326689    2571 scope.go:117] "RemoveContainer" containerID="05b72928edda8250aaff96a4d04f9909eeb8b240c42e6aad0ed1ac1b8360b416"
Jul 27 21:16:55.327013 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:16:55.326969    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b72928edda8250aaff96a4d04f9909eeb8b240c42e6aad0ed1ac1b8360b416\": container with ID starting with 05b72928edda8250aaff96a4d04f9909eeb8b240c42e6aad0ed1ac1b8360b416 not found: ID does not exist" containerID="05b72928edda8250aaff96a4d04f9909eeb8b240c42e6aad0ed1ac1b8360b416"
Jul 27 21:16:55.327013 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:55.327003    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b72928edda8250aaff96a4d04f9909eeb8b240c42e6aad0ed1ac1b8360b416"} err="failed to get container status \"05b72928edda8250aaff96a4d04f9909eeb8b240c42e6aad0ed1ac1b8360b416\": rpc error: code = NotFound desc = could not find container \"05b72928edda8250aaff96a4d04f9909eeb8b240c42e6aad0ed1ac1b8360b416\": container with ID starting with 05b72928edda8250aaff96a4d04f9909eeb8b240c42e6aad0ed1ac1b8360b416 not found: ID does not exist"
Jul 27 21:16:55.337883 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:55.337856    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596"]
Jul 27 21:16:55.341621 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:55.341603    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-070ff-5949b59c9c-h4596"]
Jul 27 21:16:55.405197 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:55.405172    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/840489b6-c02a-4575-903d-99faaf200107-proxy-tls\") pod \"error-404-isvc-d13b2-predictor-557774b68b-xp7n6\" (UID: \"840489b6-c02a-4575-903d-99faaf200107\") " pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"
Jul 27 21:16:55.407617 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:55.407600    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/840489b6-c02a-4575-903d-99faaf200107-proxy-tls\") pod \"error-404-isvc-d13b2-predictor-557774b68b-xp7n6\" (UID: \"840489b6-c02a-4575-903d-99faaf200107\") " pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"
Jul 27 21:16:55.591708 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:55.591627    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"
Jul 27 21:16:55.716905 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:55.716873    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"]
Jul 27 21:16:55.720151 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:16:55.720120    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod840489b6_c02a_4575_903d_99faaf200107.slice/crio-976bc9bf2ebd6849999b124f1f4e6704cd3932f3d025c36951087fe813d759b4 WatchSource:0}: Error finding container 976bc9bf2ebd6849999b124f1f4e6704cd3932f3d025c36951087fe813d759b4: Status 404 returned error can't find the container with id 976bc9bf2ebd6849999b124f1f4e6704cd3932f3d025c36951087fe813d759b4
Jul 27 21:16:56.324115 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:56.324071    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6" event={"ID":"840489b6-c02a-4575-903d-99faaf200107","Type":"ContainerStarted","Data":"cfa5caad5c80ae6a086c0aed31faf9221c5921c7ae1f3a91821c0c7bdee7b589"}
Jul 27 21:16:56.324593 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:56.324135    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6" event={"ID":"840489b6-c02a-4575-903d-99faaf200107","Type":"ContainerStarted","Data":"8bcafb0f6e61fa616cd01be373954afa5e3504376a0238dc4696e99376909b22"}
Jul 27 21:16:56.324593 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:56.324170    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"
Jul 27 21:16:56.324593 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:56.324199    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6" event={"ID":"840489b6-c02a-4575-903d-99faaf200107","Type":"ContainerStarted","Data":"976bc9bf2ebd6849999b124f1f4e6704cd3932f3d025c36951087fe813d759b4"}
Jul 27 21:16:56.324593 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:56.324239    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"
Jul 27 21:16:56.325633 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:56.325603    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6" podUID="840489b6-c02a-4575-903d-99faaf200107" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 27 21:16:56.343187 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:56.343145    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6" podStartSLOduration=2.343131669 podStartE2EDuration="2.343131669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:16:56.341272548 +0000 UTC m=+796.276488928" watchObservedRunningTime="2026-07-27 21:16:56.343131669 +0000 UTC m=+796.278348049"
Jul 27 21:16:56.571647 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:56.571617    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b0c3c35-819a-45b8-beab-4cb0fffeafe6" path="/var/lib/kubelet/pods/0b0c3c35-819a-45b8-beab-4cb0fffeafe6/volumes"
Jul 27 21:16:57.328109 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:16:57.328070    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6" podUID="840489b6-c02a-4575-903d-99faaf200107" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 27 21:17:00.442309 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:17:00.442277    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:17:00.510069 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:17:00.510031    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:17:02.217149 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:17:02.217060    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4" podUID="1ca8209c-921e-48e2-9de4-82a1d5d410b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 27 21:17:02.332799 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:17:02.332766    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"
Jul 27 21:17:02.333348 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:17:02.333319    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6" podUID="840489b6-c02a-4575-903d-99faaf200107" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 27 21:17:12.218016 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:17:12.217983    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"
Jul 27 21:17:12.333835 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:17:12.333786    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6" podUID="840489b6-c02a-4575-903d-99faaf200107" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 27 21:17:22.333663 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:17:22.333625    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6" podUID="840489b6-c02a-4575-903d-99faaf200107" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 27 21:17:24.726805 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:17:24.726766    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:17:25.059163 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:17:25.059121    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:17:32.333790 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:17:32.333745    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6" podUID="840489b6-c02a-4575-903d-99faaf200107" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 27 21:17:42.334105 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:17:42.334076    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"
Jul 27 21:17:54.555062 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:17:54.555027    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:17:54.878891 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:17:54.878802    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:30.114858 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:18:30.114815    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:30.217049 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:18:30.217018    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:26.452757 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:21:26.452726    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:44.887516 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:22:44.887486    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:39.391181 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.391100    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:39.466423 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.466392    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:39.539801 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.539764    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:39.575079 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.575051    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"]
Jul 27 21:25:39.575265 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.575239    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4" podUID="1ca8209c-921e-48e2-9de4-82a1d5d410b1" containerName="kserve-container" containerID="cri-o://c671a7e5838881205d5f80a3c2965ef15d8b400a670a00100239de3d8364f22a" gracePeriod=30
Jul 27 21:25:39.575394 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.575312    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4" podUID="1ca8209c-921e-48e2-9de4-82a1d5d410b1" containerName="kube-rbac-proxy" containerID="cri-o://a5550f913154f188c37a67d9c4029f12841c195ab3bed382ea65bc8292c171de" gracePeriod=30
Jul 27 21:25:39.615534 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.615505    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:39.661691 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.661631    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"]
Jul 27 21:25:39.664610 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.664596    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"
Jul 27 21:25:39.666774 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.666757    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-cb683-predictor-serving-cert\""
Jul 27 21:25:39.666889 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.666868    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-cb683-kube-rbac-proxy-sar-config\""
Jul 27 21:25:39.675627 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.675605    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"]
Jul 27 21:25:39.696648 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.696606    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:39.771920 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.771893    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/791d971c-ea7d-4d22-930f-e6d50e2b9151-proxy-tls\") pod \"error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc\" (UID: \"791d971c-ea7d-4d22-930f-e6d50e2b9151\") " pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"
Jul 27 21:25:39.772099 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.771940    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsvm2\" (UniqueName: \"kubernetes.io/projected/791d971c-ea7d-4d22-930f-e6d50e2b9151-kube-api-access-bsvm2\") pod \"error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc\" (UID: \"791d971c-ea7d-4d22-930f-e6d50e2b9151\") " pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"
Jul 27 21:25:39.772099 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.772014    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/791d971c-ea7d-4d22-930f-e6d50e2b9151-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc\" (UID: \"791d971c-ea7d-4d22-930f-e6d50e2b9151\") " pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"
Jul 27 21:25:39.872931 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.872899    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/791d971c-ea7d-4d22-930f-e6d50e2b9151-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc\" (UID: \"791d971c-ea7d-4d22-930f-e6d50e2b9151\") " pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"
Jul 27 21:25:39.873264 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.872988    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/791d971c-ea7d-4d22-930f-e6d50e2b9151-proxy-tls\") pod \"error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc\" (UID: \"791d971c-ea7d-4d22-930f-e6d50e2b9151\") " pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"
Jul 27 21:25:39.873264 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.873014    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsvm2\" (UniqueName: \"kubernetes.io/projected/791d971c-ea7d-4d22-930f-e6d50e2b9151-kube-api-access-bsvm2\") pod \"error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc\" (UID: \"791d971c-ea7d-4d22-930f-e6d50e2b9151\") " pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"
Jul 27 21:25:39.873799 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.873713    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/791d971c-ea7d-4d22-930f-e6d50e2b9151-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc\" (UID: \"791d971c-ea7d-4d22-930f-e6d50e2b9151\") " pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"
Jul 27 21:25:39.874094 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.874052    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:39.875682 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.875661    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/791d971c-ea7d-4d22-930f-e6d50e2b9151-proxy-tls\") pod \"error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc\" (UID: \"791d971c-ea7d-4d22-930f-e6d50e2b9151\") " pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"
Jul 27 21:25:39.880893 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.880873    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsvm2\" (UniqueName: \"kubernetes.io/projected/791d971c-ea7d-4d22-930f-e6d50e2b9151-kube-api-access-bsvm2\") pod \"error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc\" (UID: \"791d971c-ea7d-4d22-930f-e6d50e2b9151\") " pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"
Jul 27 21:25:39.974249 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:39.974156    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"
Jul 27 21:25:40.085361 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:40.085330    2571 generic.go:356] "Generic (PLEG): container finished" podID="1ca8209c-921e-48e2-9de4-82a1d5d410b1" containerID="a5550f913154f188c37a67d9c4029f12841c195ab3bed382ea65bc8292c171de" exitCode=2
Jul 27 21:25:40.085494 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:40.085380    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4" event={"ID":"1ca8209c-921e-48e2-9de4-82a1d5d410b1","Type":"ContainerDied","Data":"a5550f913154f188c37a67d9c4029f12841c195ab3bed382ea65bc8292c171de"}
Jul 27 21:25:40.101145 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:40.101116    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"]
Jul 27 21:25:40.103999 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:25:40.103972    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791d971c_ea7d_4d22_930f_e6d50e2b9151.slice/crio-27f0962a1c07e1ff4bddb3edddc473b7f21ea3b18d45d293f615ed9043a26bab WatchSource:0}: Error finding container 27f0962a1c07e1ff4bddb3edddc473b7f21ea3b18d45d293f615ed9043a26bab: Status 404 returned error can't find the container with id 27f0962a1c07e1ff4bddb3edddc473b7f21ea3b18d45d293f615ed9043a26bab
Jul 27 21:25:41.090691 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:41.090651    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc" event={"ID":"791d971c-ea7d-4d22-930f-e6d50e2b9151","Type":"ContainerStarted","Data":"5a82552b856969d2d46386b4f829d124cb3a7d509b7c7c24d7e309734935502f"}
Jul 27 21:25:41.091110 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:41.090722    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc" event={"ID":"791d971c-ea7d-4d22-930f-e6d50e2b9151","Type":"ContainerStarted","Data":"693b43219f02a5bcc4bf4cd9a4f567ca6b2bb0acf0352d8fcd70b0be459fc9b8"}
Jul 27 21:25:41.091110 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:41.090753    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc" event={"ID":"791d971c-ea7d-4d22-930f-e6d50e2b9151","Type":"ContainerStarted","Data":"27f0962a1c07e1ff4bddb3edddc473b7f21ea3b18d45d293f615ed9043a26bab"}
Jul 27 21:25:41.091110 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:41.090784    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"
Jul 27 21:25:41.107747 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:41.107699    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc" podStartSLOduration=2.107682666 podStartE2EDuration="2.107682666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:25:41.107120972 +0000 UTC m=+1321.042337351" watchObservedRunningTime="2026-07-27 21:25:41.107682666 +0000 UTC m=+1321.042899044"
Jul 27 21:25:42.095296 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:42.095266    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"
Jul 27 21:25:42.096597 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:42.096567    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc" podUID="791d971c-ea7d-4d22-930f-e6d50e2b9151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 27 21:25:42.213153 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:42.213112    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4" podUID="1ca8209c-921e-48e2-9de4-82a1d5d410b1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused"
Jul 27 21:25:42.217434 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:42.217409    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4" podUID="1ca8209c-921e-48e2-9de4-82a1d5d410b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 27 21:25:42.811490 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:42.811468    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"
Jul 27 21:25:42.898147 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:42.898067    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ca8209c-921e-48e2-9de4-82a1d5d410b1-kube-rbac-proxy-sar-config\") pod \"1ca8209c-921e-48e2-9de4-82a1d5d410b1\" (UID: \"1ca8209c-921e-48e2-9de4-82a1d5d410b1\") "
Jul 27 21:25:42.898147 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:42.898103    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sm9b\" (UniqueName: \"kubernetes.io/projected/1ca8209c-921e-48e2-9de4-82a1d5d410b1-kube-api-access-5sm9b\") pod \"1ca8209c-921e-48e2-9de4-82a1d5d410b1\" (UID: \"1ca8209c-921e-48e2-9de4-82a1d5d410b1\") "
Jul 27 21:25:42.898147 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:42.898122    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ca8209c-921e-48e2-9de4-82a1d5d410b1-proxy-tls\") pod \"1ca8209c-921e-48e2-9de4-82a1d5d410b1\" (UID: \"1ca8209c-921e-48e2-9de4-82a1d5d410b1\") "
Jul 27 21:25:42.898457 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:42.898436    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca8209c-921e-48e2-9de4-82a1d5d410b1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1ca8209c-921e-48e2-9de4-82a1d5d410b1" (UID: "1ca8209c-921e-48e2-9de4-82a1d5d410b1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:25:42.900259 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:42.900238    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca8209c-921e-48e2-9de4-82a1d5d410b1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ca8209c-921e-48e2-9de4-82a1d5d410b1" (UID: "1ca8209c-921e-48e2-9de4-82a1d5d410b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:25:42.900322 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:42.900252    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca8209c-921e-48e2-9de4-82a1d5d410b1-kube-api-access-5sm9b" (OuterVolumeSpecName: "kube-api-access-5sm9b") pod "1ca8209c-921e-48e2-9de4-82a1d5d410b1" (UID: "1ca8209c-921e-48e2-9de4-82a1d5d410b1"). InnerVolumeSpecName "kube-api-access-5sm9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:25:42.999322 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:42.999281    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sm9b\" (UniqueName: \"kubernetes.io/projected/1ca8209c-921e-48e2-9de4-82a1d5d410b1-kube-api-access-5sm9b\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:25:42.999322 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:42.999317    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ca8209c-921e-48e2-9de4-82a1d5d410b1-proxy-tls\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:25:42.999322 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:42.999331    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ca8209c-921e-48e2-9de4-82a1d5d410b1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:25:43.099674 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:43.099631    2571 generic.go:356] "Generic (PLEG): container finished" podID="1ca8209c-921e-48e2-9de4-82a1d5d410b1" containerID="c671a7e5838881205d5f80a3c2965ef15d8b400a670a00100239de3d8364f22a" exitCode=0
Jul 27 21:25:43.099674 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:43.099676    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"
Jul 27 21:25:43.100202 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:43.099706    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4" event={"ID":"1ca8209c-921e-48e2-9de4-82a1d5d410b1","Type":"ContainerDied","Data":"c671a7e5838881205d5f80a3c2965ef15d8b400a670a00100239de3d8364f22a"}
Jul 27 21:25:43.100202 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:43.099762    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4" event={"ID":"1ca8209c-921e-48e2-9de4-82a1d5d410b1","Type":"ContainerDied","Data":"86997674c4adfe7bc8b5c3c2afc58aa538193f60132992688e5a4444620bc2b5"}
Jul 27 21:25:43.100202 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:43.099788    2571 scope.go:117] "RemoveContainer" containerID="a5550f913154f188c37a67d9c4029f12841c195ab3bed382ea65bc8292c171de"
Jul 27 21:25:43.100371 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:43.100336    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc" podUID="791d971c-ea7d-4d22-930f-e6d50e2b9151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 27 21:25:43.113435 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:43.113414    2571 scope.go:117] "RemoveContainer" containerID="c671a7e5838881205d5f80a3c2965ef15d8b400a670a00100239de3d8364f22a"
Jul 27 21:25:43.120425 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:43.120407    2571 scope.go:117] "RemoveContainer" containerID="a5550f913154f188c37a67d9c4029f12841c195ab3bed382ea65bc8292c171de"
Jul 27 21:25:43.120676 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:25:43.120659    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5550f913154f188c37a67d9c4029f12841c195ab3bed382ea65bc8292c171de\": container with ID starting with a5550f913154f188c37a67d9c4029f12841c195ab3bed382ea65bc8292c171de not found: ID does not exist" containerID="a5550f913154f188c37a67d9c4029f12841c195ab3bed382ea65bc8292c171de"
Jul 27 21:25:43.120719 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:43.120684    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5550f913154f188c37a67d9c4029f12841c195ab3bed382ea65bc8292c171de"} err="failed to get container status \"a5550f913154f188c37a67d9c4029f12841c195ab3bed382ea65bc8292c171de\": rpc error: code = NotFound desc = could not find container \"a5550f913154f188c37a67d9c4029f12841c195ab3bed382ea65bc8292c171de\": container with ID starting with a5550f913154f188c37a67d9c4029f12841c195ab3bed382ea65bc8292c171de not found: ID does not exist"
Jul 27 21:25:43.120719 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:43.120700    2571 scope.go:117] "RemoveContainer" containerID="c671a7e5838881205d5f80a3c2965ef15d8b400a670a00100239de3d8364f22a"
Jul 27 21:25:43.120926 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:25:43.120907    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c671a7e5838881205d5f80a3c2965ef15d8b400a670a00100239de3d8364f22a\": container with ID starting with c671a7e5838881205d5f80a3c2965ef15d8b400a670a00100239de3d8364f22a not found: ID does not exist" containerID="c671a7e5838881205d5f80a3c2965ef15d8b400a670a00100239de3d8364f22a"
Jul 27 21:25:43.121014 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:43.120932    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c671a7e5838881205d5f80a3c2965ef15d8b400a670a00100239de3d8364f22a"} err="failed to get container status \"c671a7e5838881205d5f80a3c2965ef15d8b400a670a00100239de3d8364f22a\": rpc error: code = NotFound desc = could not find container \"c671a7e5838881205d5f80a3c2965ef15d8b400a670a00100239de3d8364f22a\": container with ID starting with c671a7e5838881205d5f80a3c2965ef15d8b400a670a00100239de3d8364f22a not found: ID does not exist"
Jul 27 21:25:43.123284 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:43.123265    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"]
Jul 27 21:25:43.126510 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:43.126490    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d355e-predictor-77499898bb-cghz4"]
Jul 27 21:25:44.572155 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:44.572123    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ca8209c-921e-48e2-9de4-82a1d5d410b1" path="/var/lib/kubelet/pods/1ca8209c-921e-48e2-9de4-82a1d5d410b1/volumes"
Jul 27 21:25:48.104461 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:48.104434    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"
Jul 27 21:25:48.105085 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:48.105058    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc" podUID="791d971c-ea7d-4d22-930f-e6d50e2b9151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 27 21:25:58.105872 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:25:58.105818    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc" podUID="791d971c-ea7d-4d22-930f-e6d50e2b9151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 27 21:26:08.105716 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:08.105636    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc" podUID="791d971c-ea7d-4d22-930f-e6d50e2b9151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 27 21:26:09.155291 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.155250    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:09.281661 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.281629    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:09.355374 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.355329    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"]
Jul 27 21:26:09.355631 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.355597    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6" podUID="840489b6-c02a-4575-903d-99faaf200107" containerName="kserve-container" containerID="cri-o://8bcafb0f6e61fa616cd01be373954afa5e3504376a0238dc4696e99376909b22" gracePeriod=30
Jul 27 21:26:09.355823 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.355789    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6" podUID="840489b6-c02a-4575-903d-99faaf200107" containerName="kube-rbac-proxy" containerID="cri-o://cfa5caad5c80ae6a086c0aed31faf9221c5921c7ae1f3a91821c0c7bdee7b589" gracePeriod=30
Jul 27 21:26:09.360763 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.360728    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:09.394325 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.394298    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"]
Jul 27 21:26:09.394575 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.394562    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ca8209c-921e-48e2-9de4-82a1d5d410b1" containerName="kserve-container"
Jul 27 21:26:09.394638 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.394576    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ca8209c-921e-48e2-9de4-82a1d5d410b1" containerName="kserve-container"
Jul 27 21:26:09.394638 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.394596    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ca8209c-921e-48e2-9de4-82a1d5d410b1" containerName="kube-rbac-proxy"
Jul 27 21:26:09.394638 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.394603    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ca8209c-921e-48e2-9de4-82a1d5d410b1" containerName="kube-rbac-proxy"
Jul 27 21:26:09.394775 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.394643    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ca8209c-921e-48e2-9de4-82a1d5d410b1" containerName="kserve-container"
Jul 27 21:26:09.394775 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.394651    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ca8209c-921e-48e2-9de4-82a1d5d410b1" containerName="kube-rbac-proxy"
Jul 27 21:26:09.397490 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.397477    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"
Jul 27 21:26:09.400084 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.400067    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0a095-kube-rbac-proxy-sar-config\""
Jul 27 21:26:09.400176 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.400072    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0a095-predictor-serving-cert\""
Jul 27 21:26:09.407723 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.407652    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"]
Jul 27 21:26:09.427889 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.427861    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:09.491481 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.491446    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vn8b\" (UniqueName: \"kubernetes.io/projected/dcde100a-1025-4ee1-a150-487a1086f4d3-kube-api-access-9vn8b\") pod \"error-404-isvc-0a095-predictor-745d9f4d88-hw9ff\" (UID: \"dcde100a-1025-4ee1-a150-487a1086f4d3\") " pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"
Jul 27 21:26:09.491598 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.491496    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcde100a-1025-4ee1-a150-487a1086f4d3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0a095-predictor-745d9f4d88-hw9ff\" (UID: \"dcde100a-1025-4ee1-a150-487a1086f4d3\") " pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"
Jul 27 21:26:09.491598 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.491526    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcde100a-1025-4ee1-a150-487a1086f4d3-proxy-tls\") pod \"error-404-isvc-0a095-predictor-745d9f4d88-hw9ff\" (UID: \"dcde100a-1025-4ee1-a150-487a1086f4d3\") " pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"
Jul 27 21:26:09.584438 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.584405    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:09.592717 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.592681    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vn8b\" (UniqueName: \"kubernetes.io/projected/dcde100a-1025-4ee1-a150-487a1086f4d3-kube-api-access-9vn8b\") pod \"error-404-isvc-0a095-predictor-745d9f4d88-hw9ff\" (UID: \"dcde100a-1025-4ee1-a150-487a1086f4d3\") " pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"
Jul 27 21:26:09.592885 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.592735    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcde100a-1025-4ee1-a150-487a1086f4d3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0a095-predictor-745d9f4d88-hw9ff\" (UID: \"dcde100a-1025-4ee1-a150-487a1086f4d3\") " pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"
Jul 27 21:26:09.592885 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.592865    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcde100a-1025-4ee1-a150-487a1086f4d3-proxy-tls\") pod \"error-404-isvc-0a095-predictor-745d9f4d88-hw9ff\" (UID: \"dcde100a-1025-4ee1-a150-487a1086f4d3\") " pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"
Jul 27 21:26:09.593066 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:26:09.593049    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-0a095-predictor-serving-cert: secret "error-404-isvc-0a095-predictor-serving-cert" not found
Jul 27 21:26:09.593129 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:26:09.593121    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcde100a-1025-4ee1-a150-487a1086f4d3-proxy-tls podName:dcde100a-1025-4ee1-a150-487a1086f4d3 nodeName:}" failed. No retries permitted until 2026-07-27 21:26:10.093098748 +0000 UTC m=+1350.028315118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dcde100a-1025-4ee1-a150-487a1086f4d3-proxy-tls") pod "error-404-isvc-0a095-predictor-745d9f4d88-hw9ff" (UID: "dcde100a-1025-4ee1-a150-487a1086f4d3") : secret "error-404-isvc-0a095-predictor-serving-cert" not found
Jul 27 21:26:09.593525 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.593501    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcde100a-1025-4ee1-a150-487a1086f4d3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0a095-predictor-745d9f4d88-hw9ff\" (UID: \"dcde100a-1025-4ee1-a150-487a1086f4d3\") " pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"
Jul 27 21:26:09.603946 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:09.603920    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vn8b\" (UniqueName: \"kubernetes.io/projected/dcde100a-1025-4ee1-a150-487a1086f4d3-kube-api-access-9vn8b\") pod \"error-404-isvc-0a095-predictor-745d9f4d88-hw9ff\" (UID: \"dcde100a-1025-4ee1-a150-487a1086f4d3\") " pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"
Jul 27 21:26:10.067375 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:10.067341    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:10.098532 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:10.098489    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcde100a-1025-4ee1-a150-487a1086f4d3-proxy-tls\") pod \"error-404-isvc-0a095-predictor-745d9f4d88-hw9ff\" (UID: \"dcde100a-1025-4ee1-a150-487a1086f4d3\") " pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"
Jul 27 21:26:10.101487 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:10.101455    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcde100a-1025-4ee1-a150-487a1086f4d3-proxy-tls\") pod \"error-404-isvc-0a095-predictor-745d9f4d88-hw9ff\" (UID: \"dcde100a-1025-4ee1-a150-487a1086f4d3\") " pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"
Jul 27 21:26:10.201890 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:10.201849    2571 generic.go:356] "Generic (PLEG): container finished" podID="840489b6-c02a-4575-903d-99faaf200107" containerID="cfa5caad5c80ae6a086c0aed31faf9221c5921c7ae1f3a91821c0c7bdee7b589" exitCode=2
Jul 27 21:26:10.202263 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:10.201926    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6" event={"ID":"840489b6-c02a-4575-903d-99faaf200107","Type":"ContainerDied","Data":"cfa5caad5c80ae6a086c0aed31faf9221c5921c7ae1f3a91821c0c7bdee7b589"}
Jul 27 21:26:10.309017 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:10.308981    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"
Jul 27 21:26:10.442290 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:10.442250    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"]
Jul 27 21:26:10.444714 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:26:10.444683    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcde100a_1025_4ee1_a150_487a1086f4d3.slice/crio-ea56bdbfcec8a558e96588ca9590899e0bd6590f22a6d2ac767b08943d357d83 WatchSource:0}: Error finding container ea56bdbfcec8a558e96588ca9590899e0bd6590f22a6d2ac767b08943d357d83: Status 404 returned error can't find the container with id ea56bdbfcec8a558e96588ca9590899e0bd6590f22a6d2ac767b08943d357d83
Jul 27 21:26:11.206667 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:11.206629    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff" event={"ID":"dcde100a-1025-4ee1-a150-487a1086f4d3","Type":"ContainerStarted","Data":"932de09058244bd06e046703cb48d53965f451ee3e8678f35f99fbaff494fbc0"}
Jul 27 21:26:11.207122 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:11.206687    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff" event={"ID":"dcde100a-1025-4ee1-a150-487a1086f4d3","Type":"ContainerStarted","Data":"0493113b6f0a3c7be823456b6985de8e44e0c13e9105d9ed89c3024cc7315446"}
Jul 27 21:26:11.207122 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:11.206709    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff" event={"ID":"dcde100a-1025-4ee1-a150-487a1086f4d3","Type":"ContainerStarted","Data":"ea56bdbfcec8a558e96588ca9590899e0bd6590f22a6d2ac767b08943d357d83"}
Jul 27 21:26:11.207122 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:11.206730    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"
Jul 27 21:26:11.207122 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:11.206860    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"
Jul 27 21:26:11.208204 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:11.208178    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff" podUID="dcde100a-1025-4ee1-a150-487a1086f4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 27 21:26:11.225115 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:11.225070    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff" podStartSLOduration=2.225054197 podStartE2EDuration="2.225054197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:26:11.223814036 +0000 UTC m=+1351.159030416" watchObservedRunningTime="2026-07-27 21:26:11.225054197 +0000 UTC m=+1351.160270576"
Jul 27 21:26:12.210278 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:12.210235    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff" podUID="dcde100a-1025-4ee1-a150-487a1086f4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 27 21:26:12.329347 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:12.329311    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6" podUID="840489b6-c02a-4575-903d-99faaf200107" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
Jul 27 21:26:12.333473 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:12.333452    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6" podUID="840489b6-c02a-4575-903d-99faaf200107" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 27 21:26:12.590157 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:12.590134    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"
Jul 27 21:26:12.616597 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:12.616566    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/840489b6-c02a-4575-903d-99faaf200107-proxy-tls\") pod \"840489b6-c02a-4575-903d-99faaf200107\" (UID: \"840489b6-c02a-4575-903d-99faaf200107\") "
Jul 27 21:26:12.616758 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:12.616649    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/840489b6-c02a-4575-903d-99faaf200107-kube-rbac-proxy-sar-config\") pod \"840489b6-c02a-4575-903d-99faaf200107\" (UID: \"840489b6-c02a-4575-903d-99faaf200107\") "
Jul 27 21:26:12.616758 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:12.616703    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vff7v\" (UniqueName: \"kubernetes.io/projected/840489b6-c02a-4575-903d-99faaf200107-kube-api-access-vff7v\") pod \"840489b6-c02a-4575-903d-99faaf200107\" (UID: \"840489b6-c02a-4575-903d-99faaf200107\") "
Jul 27 21:26:12.616972 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:12.616933    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/840489b6-c02a-4575-903d-99faaf200107-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "840489b6-c02a-4575-903d-99faaf200107" (UID: "840489b6-c02a-4575-903d-99faaf200107"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:26:12.618811 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:12.618783    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/840489b6-c02a-4575-903d-99faaf200107-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "840489b6-c02a-4575-903d-99faaf200107" (UID: "840489b6-c02a-4575-903d-99faaf200107"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:26:12.618912 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:12.618876    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/840489b6-c02a-4575-903d-99faaf200107-kube-api-access-vff7v" (OuterVolumeSpecName: "kube-api-access-vff7v") pod "840489b6-c02a-4575-903d-99faaf200107" (UID: "840489b6-c02a-4575-903d-99faaf200107"). InnerVolumeSpecName "kube-api-access-vff7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:26:12.717629 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:12.717598    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/840489b6-c02a-4575-903d-99faaf200107-proxy-tls\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:26:12.717629 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:12.717622    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/840489b6-c02a-4575-903d-99faaf200107-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:26:12.717629 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:12.717631    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vff7v\" (UniqueName: \"kubernetes.io/projected/840489b6-c02a-4575-903d-99faaf200107-kube-api-access-vff7v\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:26:13.214526 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:13.214490    2571 generic.go:356] "Generic (PLEG): container finished" podID="840489b6-c02a-4575-903d-99faaf200107" containerID="8bcafb0f6e61fa616cd01be373954afa5e3504376a0238dc4696e99376909b22" exitCode=0
Jul 27 21:26:13.214940 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:13.214537    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"
Jul 27 21:26:13.214940 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:13.214554    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6" event={"ID":"840489b6-c02a-4575-903d-99faaf200107","Type":"ContainerDied","Data":"8bcafb0f6e61fa616cd01be373954afa5e3504376a0238dc4696e99376909b22"}
Jul 27 21:26:13.214940 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:13.214605    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6" event={"ID":"840489b6-c02a-4575-903d-99faaf200107","Type":"ContainerDied","Data":"976bc9bf2ebd6849999b124f1f4e6704cd3932f3d025c36951087fe813d759b4"}
Jul 27 21:26:13.214940 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:13.214632    2571 scope.go:117] "RemoveContainer" containerID="cfa5caad5c80ae6a086c0aed31faf9221c5921c7ae1f3a91821c0c7bdee7b589"
Jul 27 21:26:13.222867 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:13.222838    2571 scope.go:117] "RemoveContainer" containerID="8bcafb0f6e61fa616cd01be373954afa5e3504376a0238dc4696e99376909b22"
Jul 27 21:26:13.231483 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:13.231458    2571 scope.go:117] "RemoveContainer" containerID="cfa5caad5c80ae6a086c0aed31faf9221c5921c7ae1f3a91821c0c7bdee7b589"
Jul 27 21:26:13.231919 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:26:13.231902    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfa5caad5c80ae6a086c0aed31faf9221c5921c7ae1f3a91821c0c7bdee7b589\": container with ID starting with cfa5caad5c80ae6a086c0aed31faf9221c5921c7ae1f3a91821c0c7bdee7b589 not found: ID does not exist" containerID="cfa5caad5c80ae6a086c0aed31faf9221c5921c7ae1f3a91821c0c7bdee7b589"
Jul 27 21:26:13.231980 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:13.231930    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfa5caad5c80ae6a086c0aed31faf9221c5921c7ae1f3a91821c0c7bdee7b589"} err="failed to get container status \"cfa5caad5c80ae6a086c0aed31faf9221c5921c7ae1f3a91821c0c7bdee7b589\": rpc error: code = NotFound desc = could not find container \"cfa5caad5c80ae6a086c0aed31faf9221c5921c7ae1f3a91821c0c7bdee7b589\": container with ID starting with cfa5caad5c80ae6a086c0aed31faf9221c5921c7ae1f3a91821c0c7bdee7b589 not found: ID does not exist"
Jul 27 21:26:13.231980 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:13.231948    2571 scope.go:117] "RemoveContainer" containerID="8bcafb0f6e61fa616cd01be373954afa5e3504376a0238dc4696e99376909b22"
Jul 27 21:26:13.232239 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:26:13.232224    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bcafb0f6e61fa616cd01be373954afa5e3504376a0238dc4696e99376909b22\": container with ID starting with 8bcafb0f6e61fa616cd01be373954afa5e3504376a0238dc4696e99376909b22 not found: ID does not exist" containerID="8bcafb0f6e61fa616cd01be373954afa5e3504376a0238dc4696e99376909b22"
Jul 27 21:26:13.232296 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:13.232242    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bcafb0f6e61fa616cd01be373954afa5e3504376a0238dc4696e99376909b22"} err="failed to get container status \"8bcafb0f6e61fa616cd01be373954afa5e3504376a0238dc4696e99376909b22\": rpc error: code = NotFound desc = could not find container \"8bcafb0f6e61fa616cd01be373954afa5e3504376a0238dc4696e99376909b22\": container with ID starting with 8bcafb0f6e61fa616cd01be373954afa5e3504376a0238dc4696e99376909b22 not found: ID does not exist"
Jul 27 21:26:13.237035 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:13.237009    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"]
Jul 27 21:26:13.242481 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:13.242456    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d13b2-predictor-557774b68b-xp7n6"]
Jul 27 21:26:14.571729 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:14.571698    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="840489b6-c02a-4575-903d-99faaf200107" path="/var/lib/kubelet/pods/840489b6-c02a-4575-903d-99faaf200107/volumes"
Jul 27 21:26:17.215297 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:17.215273    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"
Jul 27 21:26:17.215741 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:17.215714    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff" podUID="dcde100a-1025-4ee1-a150-487a1086f4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 27 21:26:18.105716 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:18.105681    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc" podUID="791d971c-ea7d-4d22-930f-e6d50e2b9151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 27 21:26:27.216595 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:27.216553    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff" podUID="dcde100a-1025-4ee1-a150-487a1086f4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 27 21:26:28.106538 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:28.106511    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"
Jul 27 21:26:37.216375 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:37.216334    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff" podUID="dcde100a-1025-4ee1-a150-487a1086f4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 27 21:26:39.626695 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:39.626659    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:40.902135 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:40.902085    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:47.216795 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:47.216750    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff" podUID="dcde100a-1025-4ee1-a150-487a1086f4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 27 21:26:49.703621 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:49.703576    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:49.792935 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:49.792909    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"]
Jul 27 21:26:49.793200 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:49.793171    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc" podUID="791d971c-ea7d-4d22-930f-e6d50e2b9151" containerName="kserve-container" containerID="cri-o://693b43219f02a5bcc4bf4cd9a4f567ca6b2bb0acf0352d8fcd70b0be459fc9b8" gracePeriod=30
Jul 27 21:26:49.793324 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:49.793249    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc" podUID="791d971c-ea7d-4d22-930f-e6d50e2b9151" containerName="kube-rbac-proxy" containerID="cri-o://5a82552b856969d2d46386b4f829d124cb3a7d509b7c7c24d7e309734935502f" gracePeriod=30
Jul 27 21:26:49.826055 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:49.825943    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:49.911325 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:49.911298    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:49.989487 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:49.989411    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"]
Jul 27 21:26:49.989746 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:49.989729    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="840489b6-c02a-4575-903d-99faaf200107" containerName="kube-rbac-proxy"
Jul 27 21:26:49.989825 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:49.989749    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="840489b6-c02a-4575-903d-99faaf200107" containerName="kube-rbac-proxy"
Jul 27 21:26:49.989825 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:49.989769    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="840489b6-c02a-4575-903d-99faaf200107" containerName="kserve-container"
Jul 27 21:26:49.989825 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:49.989777    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="840489b6-c02a-4575-903d-99faaf200107" containerName="kserve-container"
Jul 27 21:26:49.990012 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:49.989869    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="840489b6-c02a-4575-903d-99faaf200107" containerName="kube-rbac-proxy"
Jul 27 21:26:49.990012 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:49.989885    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="840489b6-c02a-4575-903d-99faaf200107" containerName="kserve-container"
Jul 27 21:26:49.992836 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:49.992816    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"
Jul 27 21:26:49.995799 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:49.995761    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e26c5-predictor-serving-cert\""
Jul 27 21:26:49.995799 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:49.995793    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e26c5-kube-rbac-proxy-sar-config\""
Jul 27 21:26:50.001480 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:50.001454    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:50.010111 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:50.010089    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"]
Jul 27 21:26:50.072127 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:50.072094    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vllgs\" (UniqueName: \"kubernetes.io/projected/e12f0e51-4d30-47ec-9730-7f1f552d3461-kube-api-access-vllgs\") pod \"error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7\" (UID: \"e12f0e51-4d30-47ec-9730-7f1f552d3461\") " pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"
Jul 27 21:26:50.072269 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:50.072155    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e12f0e51-4d30-47ec-9730-7f1f552d3461-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7\" (UID: \"e12f0e51-4d30-47ec-9730-7f1f552d3461\") " pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"
Jul 27 21:26:50.072269 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:50.072241    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e12f0e51-4d30-47ec-9730-7f1f552d3461-proxy-tls\") pod \"error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7\" (UID: \"e12f0e51-4d30-47ec-9730-7f1f552d3461\") " pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"
Jul 27 21:26:50.083643 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:50.083619    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:50.173449 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:50.173424    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e12f0e51-4d30-47ec-9730-7f1f552d3461-proxy-tls\") pod \"error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7\" (UID: \"e12f0e51-4d30-47ec-9730-7f1f552d3461\") " pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"
Jul 27 21:26:50.173564 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:50.173478    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vllgs\" (UniqueName: \"kubernetes.io/projected/e12f0e51-4d30-47ec-9730-7f1f552d3461-kube-api-access-vllgs\") pod \"error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7\" (UID: \"e12f0e51-4d30-47ec-9730-7f1f552d3461\") " pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"
Jul 27 21:26:50.173564 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:50.173529    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e12f0e51-4d30-47ec-9730-7f1f552d3461-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7\" (UID: \"e12f0e51-4d30-47ec-9730-7f1f552d3461\") " pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"
Jul 27 21:26:50.173657 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:26:50.173582    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-serving-cert: secret "error-404-isvc-e26c5-predictor-serving-cert" not found
Jul 27 21:26:50.173657 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:26:50.173645    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12f0e51-4d30-47ec-9730-7f1f552d3461-proxy-tls podName:e12f0e51-4d30-47ec-9730-7f1f552d3461 nodeName:}" failed. No retries permitted until 2026-07-27 21:26:50.673625609 +0000 UTC m=+1390.608841980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e12f0e51-4d30-47ec-9730-7f1f552d3461-proxy-tls") pod "error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7" (UID: "e12f0e51-4d30-47ec-9730-7f1f552d3461") : secret "error-404-isvc-e26c5-predictor-serving-cert" not found
Jul 27 21:26:50.174141 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:50.174126    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e12f0e51-4d30-47ec-9730-7f1f552d3461-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7\" (UID: \"e12f0e51-4d30-47ec-9730-7f1f552d3461\") " pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"
Jul 27 21:26:50.182074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:50.182046    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vllgs\" (UniqueName: \"kubernetes.io/projected/e12f0e51-4d30-47ec-9730-7f1f552d3461-kube-api-access-vllgs\") pod \"error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7\" (UID: \"e12f0e51-4d30-47ec-9730-7f1f552d3461\") " pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"
Jul 27 21:26:50.296641 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:50.296607    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:50.341991 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:50.341941    2571 generic.go:356] "Generic (PLEG): container finished" podID="791d971c-ea7d-4d22-930f-e6d50e2b9151" containerID="5a82552b856969d2d46386b4f829d124cb3a7d509b7c7c24d7e309734935502f" exitCode=2
Jul 27 21:26:50.341991 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:50.341982    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc" event={"ID":"791d971c-ea7d-4d22-930f-e6d50e2b9151","Type":"ContainerDied","Data":"5a82552b856969d2d46386b4f829d124cb3a7d509b7c7c24d7e309734935502f"}
Jul 27 21:26:50.677803 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:50.677696    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e12f0e51-4d30-47ec-9730-7f1f552d3461-proxy-tls\") pod \"error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7\" (UID: \"e12f0e51-4d30-47ec-9730-7f1f552d3461\") " pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"
Jul 27 21:26:50.680280 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:50.680260    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e12f0e51-4d30-47ec-9730-7f1f552d3461-proxy-tls\") pod \"error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7\" (UID: \"e12f0e51-4d30-47ec-9730-7f1f552d3461\") " pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"
Jul 27 21:26:50.903720 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:50.903687    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"
Jul 27 21:26:51.028592 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:51.028552    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"]
Jul 27 21:26:51.032206 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:26:51.032183    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12f0e51_4d30_47ec_9730_7f1f552d3461.slice/crio-5e40d6f27f7421bf1cee4c2a1b5f8fe02b5e3bbfb7838929d7ad97253e3b4718 WatchSource:0}: Error finding container 5e40d6f27f7421bf1cee4c2a1b5f8fe02b5e3bbfb7838929d7ad97253e3b4718: Status 404 returned error can't find the container with id 5e40d6f27f7421bf1cee4c2a1b5f8fe02b5e3bbfb7838929d7ad97253e3b4718
Jul 27 21:26:51.347506 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:51.347461    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7" event={"ID":"e12f0e51-4d30-47ec-9730-7f1f552d3461","Type":"ContainerStarted","Data":"175076e4ee812964cae16abc2828c5412e9de2917cf4e13ea885631ab67d441b"}
Jul 27 21:26:51.347686 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:51.347532    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"
Jul 27 21:26:51.347686 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:51.347553    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7" event={"ID":"e12f0e51-4d30-47ec-9730-7f1f552d3461","Type":"ContainerStarted","Data":"29714127e0154cae514aba4a2a81937df30b97377a2b56a6b55c16d5f9b04efd"}
Jul 27 21:26:51.347686 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:51.347571    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7" event={"ID":"e12f0e51-4d30-47ec-9730-7f1f552d3461","Type":"ContainerStarted","Data":"5e40d6f27f7421bf1cee4c2a1b5f8fe02b5e3bbfb7838929d7ad97253e3b4718"}
Jul 27 21:26:51.347686 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:51.347597    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"
Jul 27 21:26:51.349084 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:51.349057    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7" podUID="e12f0e51-4d30-47ec-9730-7f1f552d3461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 27 21:26:51.363997 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:51.363929    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7" podStartSLOduration=2.363892194 podStartE2EDuration="2.363892194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:26:51.363230643 +0000 UTC m=+1391.298447021" watchObservedRunningTime="2026-07-27 21:26:51.363892194 +0000 UTC m=+1391.299108576"
Jul 27 21:26:52.351186 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:52.351148    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7" podUID="e12f0e51-4d30-47ec-9730-7f1f552d3461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 27 21:26:52.936804 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:52.936782    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"
Jul 27 21:26:52.997696 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:52.997656    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/791d971c-ea7d-4d22-930f-e6d50e2b9151-proxy-tls\") pod \"791d971c-ea7d-4d22-930f-e6d50e2b9151\" (UID: \"791d971c-ea7d-4d22-930f-e6d50e2b9151\") "
Jul 27 21:26:52.997889 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:52.997717    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/791d971c-ea7d-4d22-930f-e6d50e2b9151-kube-rbac-proxy-sar-config\") pod \"791d971c-ea7d-4d22-930f-e6d50e2b9151\" (UID: \"791d971c-ea7d-4d22-930f-e6d50e2b9151\") "
Jul 27 21:26:52.997889 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:52.997768    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsvm2\" (UniqueName: \"kubernetes.io/projected/791d971c-ea7d-4d22-930f-e6d50e2b9151-kube-api-access-bsvm2\") pod \"791d971c-ea7d-4d22-930f-e6d50e2b9151\" (UID: \"791d971c-ea7d-4d22-930f-e6d50e2b9151\") "
Jul 27 21:26:52.998161 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:52.998130    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/791d971c-ea7d-4d22-930f-e6d50e2b9151-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "791d971c-ea7d-4d22-930f-e6d50e2b9151" (UID: "791d971c-ea7d-4d22-930f-e6d50e2b9151"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:26:53.000022 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:52.999987    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/791d971c-ea7d-4d22-930f-e6d50e2b9151-kube-api-access-bsvm2" (OuterVolumeSpecName: "kube-api-access-bsvm2") pod "791d971c-ea7d-4d22-930f-e6d50e2b9151" (UID: "791d971c-ea7d-4d22-930f-e6d50e2b9151"). InnerVolumeSpecName "kube-api-access-bsvm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:26:53.000022 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:53.000000    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/791d971c-ea7d-4d22-930f-e6d50e2b9151-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "791d971c-ea7d-4d22-930f-e6d50e2b9151" (UID: "791d971c-ea7d-4d22-930f-e6d50e2b9151"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:26:53.098312 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:53.098233    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/791d971c-ea7d-4d22-930f-e6d50e2b9151-proxy-tls\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:26:53.098312 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:53.098263    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/791d971c-ea7d-4d22-930f-e6d50e2b9151-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:26:53.098312 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:53.098273    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsvm2\" (UniqueName: \"kubernetes.io/projected/791d971c-ea7d-4d22-930f-e6d50e2b9151-kube-api-access-bsvm2\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:26:53.355269 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:53.355182    2571 generic.go:356] "Generic (PLEG): container finished" podID="791d971c-ea7d-4d22-930f-e6d50e2b9151" containerID="693b43219f02a5bcc4bf4cd9a4f567ca6b2bb0acf0352d8fcd70b0be459fc9b8" exitCode=0
Jul 27 21:26:53.355269 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:53.355236    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"
Jul 27 21:26:53.355785 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:53.355235    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc" event={"ID":"791d971c-ea7d-4d22-930f-e6d50e2b9151","Type":"ContainerDied","Data":"693b43219f02a5bcc4bf4cd9a4f567ca6b2bb0acf0352d8fcd70b0be459fc9b8"}
Jul 27 21:26:53.355785 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:53.355392    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc" event={"ID":"791d971c-ea7d-4d22-930f-e6d50e2b9151","Type":"ContainerDied","Data":"27f0962a1c07e1ff4bddb3edddc473b7f21ea3b18d45d293f615ed9043a26bab"}
Jul 27 21:26:53.355785 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:53.355426    2571 scope.go:117] "RemoveContainer" containerID="5a82552b856969d2d46386b4f829d124cb3a7d509b7c7c24d7e309734935502f"
Jul 27 21:26:53.363749 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:53.363726    2571 scope.go:117] "RemoveContainer" containerID="693b43219f02a5bcc4bf4cd9a4f567ca6b2bb0acf0352d8fcd70b0be459fc9b8"
Jul 27 21:26:53.371087 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:53.371070    2571 scope.go:117] "RemoveContainer" containerID="5a82552b856969d2d46386b4f829d124cb3a7d509b7c7c24d7e309734935502f"
Jul 27 21:26:53.371343 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:26:53.371325    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a82552b856969d2d46386b4f829d124cb3a7d509b7c7c24d7e309734935502f\": container with ID starting with 5a82552b856969d2d46386b4f829d124cb3a7d509b7c7c24d7e309734935502f not found: ID does not exist" containerID="5a82552b856969d2d46386b4f829d124cb3a7d509b7c7c24d7e309734935502f"
Jul 27 21:26:53.371392 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:53.371353    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a82552b856969d2d46386b4f829d124cb3a7d509b7c7c24d7e309734935502f"} err="failed to get container status \"5a82552b856969d2d46386b4f829d124cb3a7d509b7c7c24d7e309734935502f\": rpc error: code = NotFound desc = could not find container \"5a82552b856969d2d46386b4f829d124cb3a7d509b7c7c24d7e309734935502f\": container with ID starting with 5a82552b856969d2d46386b4f829d124cb3a7d509b7c7c24d7e309734935502f not found: ID does not exist"
Jul 27 21:26:53.371392 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:53.371371    2571 scope.go:117] "RemoveContainer" containerID="693b43219f02a5bcc4bf4cd9a4f567ca6b2bb0acf0352d8fcd70b0be459fc9b8"
Jul 27 21:26:53.371586 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:26:53.371573    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"693b43219f02a5bcc4bf4cd9a4f567ca6b2bb0acf0352d8fcd70b0be459fc9b8\": container with ID starting with 693b43219f02a5bcc4bf4cd9a4f567ca6b2bb0acf0352d8fcd70b0be459fc9b8 not found: ID does not exist" containerID="693b43219f02a5bcc4bf4cd9a4f567ca6b2bb0acf0352d8fcd70b0be459fc9b8"
Jul 27 21:26:53.371633 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:53.371590    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"693b43219f02a5bcc4bf4cd9a4f567ca6b2bb0acf0352d8fcd70b0be459fc9b8"} err="failed to get container status \"693b43219f02a5bcc4bf4cd9a4f567ca6b2bb0acf0352d8fcd70b0be459fc9b8\": rpc error: code = NotFound desc = could not find container \"693b43219f02a5bcc4bf4cd9a4f567ca6b2bb0acf0352d8fcd70b0be459fc9b8\": container with ID starting with 693b43219f02a5bcc4bf4cd9a4f567ca6b2bb0acf0352d8fcd70b0be459fc9b8 not found: ID does not exist"
Jul 27 21:26:53.375167 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:53.375147    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"]
Jul 27 21:26:53.381072 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:53.381054    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb683-predictor-84d5bf79d6-bz4dc"]
Jul 27 21:26:54.572148 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:54.572116    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="791d971c-ea7d-4d22-930f-e6d50e2b9151" path="/var/lib/kubelet/pods/791d971c-ea7d-4d22-930f-e6d50e2b9151/volumes"
Jul 27 21:26:57.216509 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:57.216484    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"
Jul 27 21:26:57.356128 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:57.356102    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"
Jul 27 21:26:57.356570 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:26:57.356545    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7" podUID="e12f0e51-4d30-47ec-9730-7f1f552d3461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 27 21:27:07.357054 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:07.357018    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7" podUID="e12f0e51-4d30-47ec-9730-7f1f552d3461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 27 21:27:09.329843 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:09.329807    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:09.484278 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:09.484246    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:17.356737 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:17.356699    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7" podUID="e12f0e51-4d30-47ec-9730-7f1f552d3461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 27 21:27:19.415860 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.415828    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:19.535358 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.535324    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:19.556904 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.556874    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"]
Jul 27 21:27:19.557150 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.557126    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff" podUID="dcde100a-1025-4ee1-a150-487a1086f4d3" containerName="kserve-container" containerID="cri-o://0493113b6f0a3c7be823456b6985de8e44e0c13e9105d9ed89c3024cc7315446" gracePeriod=30
Jul 27 21:27:19.557228 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.557132    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff" podUID="dcde100a-1025-4ee1-a150-487a1086f4d3" containerName="kube-rbac-proxy" containerID="cri-o://932de09058244bd06e046703cb48d53965f451ee3e8678f35f99fbaff494fbc0" gracePeriod=30
Jul 27 21:27:19.601343 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.601305    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:19.672842 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.672818    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"]
Jul 27 21:27:19.673112 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.673098    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791d971c-ea7d-4d22-930f-e6d50e2b9151" containerName="kserve-container"
Jul 27 21:27:19.673181 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.673114    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="791d971c-ea7d-4d22-930f-e6d50e2b9151" containerName="kserve-container"
Jul 27 21:27:19.673181 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.673126    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791d971c-ea7d-4d22-930f-e6d50e2b9151" containerName="kube-rbac-proxy"
Jul 27 21:27:19.673181 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.673131    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="791d971c-ea7d-4d22-930f-e6d50e2b9151" containerName="kube-rbac-proxy"
Jul 27 21:27:19.673181 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.673175    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="791d971c-ea7d-4d22-930f-e6d50e2b9151" containerName="kserve-container"
Jul 27 21:27:19.673181 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.673183    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="791d971c-ea7d-4d22-930f-e6d50e2b9151" containerName="kube-rbac-proxy"
Jul 27 21:27:19.675899 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.675882    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"
Jul 27 21:27:19.679076 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.679052    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0f5d7-predictor-serving-cert\""
Jul 27 21:27:19.679195 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.679113    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0f5d7-kube-rbac-proxy-sar-config\""
Jul 27 21:27:19.680910 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.680880    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:19.688177 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.688159    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"]
Jul 27 21:27:19.791586 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.791543    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwrtt\" (UniqueName: \"kubernetes.io/projected/eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d-kube-api-access-jwrtt\") pod \"error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq\" (UID: \"eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d\") " pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"
Jul 27 21:27:19.791586 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.791585    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq\" (UID: \"eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d\") " pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"
Jul 27 21:27:19.791811 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.791673    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d-proxy-tls\") pod \"error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq\" (UID: \"eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d\") " pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"
Jul 27 21:27:19.892265 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.892234    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwrtt\" (UniqueName: \"kubernetes.io/projected/eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d-kube-api-access-jwrtt\") pod \"error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq\" (UID: \"eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d\") " pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"
Jul 27 21:27:19.892265 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.892268    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq\" (UID: \"eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d\") " pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"
Jul 27 21:27:19.892508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.892346    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d-proxy-tls\") pod \"error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq\" (UID: \"eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d\") " pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"
Jul 27 21:27:19.893115 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.893088    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq\" (UID: \"eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d\") " pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"
Jul 27 21:27:19.894999 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.894973    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d-proxy-tls\") pod \"error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq\" (UID: \"eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d\") " pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"
Jul 27 21:27:19.896911 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.896885    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:19.901330 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.901312    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwrtt\" (UniqueName: \"kubernetes.io/projected/eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d-kube-api-access-jwrtt\") pod \"error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq\" (UID: \"eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d\") " pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"
Jul 27 21:27:19.985502 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:19.985400    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"
Jul 27 21:27:20.078178 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:20.078147    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:20.120300 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:20.120271    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"]
Jul 27 21:27:20.124067 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:27:20.124040    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba7c1dc_88e9_4f1a_83ea_adb3be7f8d6d.slice/crio-c14368f44f81fa3c7025743fe683ee325881c7cb7e79cafdf6625551bfae3e70 WatchSource:0}: Error finding container c14368f44f81fa3c7025743fe683ee325881c7cb7e79cafdf6625551bfae3e70: Status 404 returned error can't find the container with id c14368f44f81fa3c7025743fe683ee325881c7cb7e79cafdf6625551bfae3e70
Jul 27 21:27:20.447091 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:20.447060    2571 generic.go:356] "Generic (PLEG): container finished" podID="dcde100a-1025-4ee1-a150-487a1086f4d3" containerID="932de09058244bd06e046703cb48d53965f451ee3e8678f35f99fbaff494fbc0" exitCode=2
Jul 27 21:27:20.447522 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:20.447141    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff" event={"ID":"dcde100a-1025-4ee1-a150-487a1086f4d3","Type":"ContainerDied","Data":"932de09058244bd06e046703cb48d53965f451ee3e8678f35f99fbaff494fbc0"}
Jul 27 21:27:20.448786 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:20.448766    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq" event={"ID":"eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d","Type":"ContainerStarted","Data":"1af79d0721623fe8dce313488d1a82cfad38f88ac757f46f4ceb85749330cc43"}
Jul 27 21:27:20.448896 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:20.448815    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq" event={"ID":"eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d","Type":"ContainerStarted","Data":"074504426d783abccaa8b33b1c27578b6cdb87e79c37a84cd89ccca3b323923b"}
Jul 27 21:27:20.448896 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:20.448838    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq" event={"ID":"eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d","Type":"ContainerStarted","Data":"c14368f44f81fa3c7025743fe683ee325881c7cb7e79cafdf6625551bfae3e70"}
Jul 27 21:27:20.448983 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:20.448937    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"
Jul 27 21:27:20.468605 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:20.468565    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq" podStartSLOduration=1.468552563 podStartE2EDuration="1.468552563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:27:20.466562334 +0000 UTC m=+1420.401778713" watchObservedRunningTime="2026-07-27 21:27:20.468552563 +0000 UTC m=+1420.403768946"
Jul 27 21:27:21.453039 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:21.453008    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"
Jul 27 21:27:21.454229 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:21.454206    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq" podUID="eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 27 21:27:22.211003 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:22.210933    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff" podUID="dcde100a-1025-4ee1-a150-487a1086f4d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Jul 27 21:27:22.456218 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:22.456179    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq" podUID="eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 27 21:27:22.894112 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:22.894082    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"
Jul 27 21:27:23.020343 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:23.020311    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcde100a-1025-4ee1-a150-487a1086f4d3-kube-rbac-proxy-sar-config\") pod \"dcde100a-1025-4ee1-a150-487a1086f4d3\" (UID: \"dcde100a-1025-4ee1-a150-487a1086f4d3\") "
Jul 27 21:27:23.020526 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:23.020368    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcde100a-1025-4ee1-a150-487a1086f4d3-proxy-tls\") pod \"dcde100a-1025-4ee1-a150-487a1086f4d3\" (UID: \"dcde100a-1025-4ee1-a150-487a1086f4d3\") "
Jul 27 21:27:23.020526 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:23.020399    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vn8b\" (UniqueName: \"kubernetes.io/projected/dcde100a-1025-4ee1-a150-487a1086f4d3-kube-api-access-9vn8b\") pod \"dcde100a-1025-4ee1-a150-487a1086f4d3\" (UID: \"dcde100a-1025-4ee1-a150-487a1086f4d3\") "
Jul 27 21:27:23.020684 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:23.020659    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcde100a-1025-4ee1-a150-487a1086f4d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dcde100a-1025-4ee1-a150-487a1086f4d3" (UID: "dcde100a-1025-4ee1-a150-487a1086f4d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:27:23.022522 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:23.022493    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcde100a-1025-4ee1-a150-487a1086f4d3-kube-api-access-9vn8b" (OuterVolumeSpecName: "kube-api-access-9vn8b") pod "dcde100a-1025-4ee1-a150-487a1086f4d3" (UID: "dcde100a-1025-4ee1-a150-487a1086f4d3"). InnerVolumeSpecName "kube-api-access-9vn8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:27:23.022678 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:23.022658    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcde100a-1025-4ee1-a150-487a1086f4d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dcde100a-1025-4ee1-a150-487a1086f4d3" (UID: "dcde100a-1025-4ee1-a150-487a1086f4d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:27:23.121255 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:23.121221    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcde100a-1025-4ee1-a150-487a1086f4d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:27:23.121255 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:23.121249    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcde100a-1025-4ee1-a150-487a1086f4d3-proxy-tls\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:27:23.121255 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:23.121262    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vn8b\" (UniqueName: \"kubernetes.io/projected/dcde100a-1025-4ee1-a150-487a1086f4d3-kube-api-access-9vn8b\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:27:23.461019 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:23.460933    2571 generic.go:356] "Generic (PLEG): container finished" podID="dcde100a-1025-4ee1-a150-487a1086f4d3" containerID="0493113b6f0a3c7be823456b6985de8e44e0c13e9105d9ed89c3024cc7315446" exitCode=0
Jul 27 21:27:23.461019 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:23.461005    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"
Jul 27 21:27:23.461477 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:23.461011    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff" event={"ID":"dcde100a-1025-4ee1-a150-487a1086f4d3","Type":"ContainerDied","Data":"0493113b6f0a3c7be823456b6985de8e44e0c13e9105d9ed89c3024cc7315446"}
Jul 27 21:27:23.461477 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:23.461069    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff" event={"ID":"dcde100a-1025-4ee1-a150-487a1086f4d3","Type":"ContainerDied","Data":"ea56bdbfcec8a558e96588ca9590899e0bd6590f22a6d2ac767b08943d357d83"}
Jul 27 21:27:23.461477 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:23.461096    2571 scope.go:117] "RemoveContainer" containerID="932de09058244bd06e046703cb48d53965f451ee3e8678f35f99fbaff494fbc0"
Jul 27 21:27:23.469342 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:23.469324    2571 scope.go:117] "RemoveContainer" containerID="0493113b6f0a3c7be823456b6985de8e44e0c13e9105d9ed89c3024cc7315446"
Jul 27 21:27:23.476389 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:23.476375    2571 scope.go:117] "RemoveContainer" containerID="932de09058244bd06e046703cb48d53965f451ee3e8678f35f99fbaff494fbc0"
Jul 27 21:27:23.476643 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:27:23.476626    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"932de09058244bd06e046703cb48d53965f451ee3e8678f35f99fbaff494fbc0\": container with ID starting with 932de09058244bd06e046703cb48d53965f451ee3e8678f35f99fbaff494fbc0 not found: ID does not exist" containerID="932de09058244bd06e046703cb48d53965f451ee3e8678f35f99fbaff494fbc0"
Jul 27 21:27:23.476687 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:23.476652    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"932de09058244bd06e046703cb48d53965f451ee3e8678f35f99fbaff494fbc0"} err="failed to get container status \"932de09058244bd06e046703cb48d53965f451ee3e8678f35f99fbaff494fbc0\": rpc error: code = NotFound desc = could not find container \"932de09058244bd06e046703cb48d53965f451ee3e8678f35f99fbaff494fbc0\": container with ID starting with 932de09058244bd06e046703cb48d53965f451ee3e8678f35f99fbaff494fbc0 not found: ID does not exist"
Jul 27 21:27:23.476687 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:23.476670    2571 scope.go:117] "RemoveContainer" containerID="0493113b6f0a3c7be823456b6985de8e44e0c13e9105d9ed89c3024cc7315446"
Jul 27 21:27:23.476900 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:27:23.476882    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0493113b6f0a3c7be823456b6985de8e44e0c13e9105d9ed89c3024cc7315446\": container with ID starting with 0493113b6f0a3c7be823456b6985de8e44e0c13e9105d9ed89c3024cc7315446 not found: ID does not exist" containerID="0493113b6f0a3c7be823456b6985de8e44e0c13e9105d9ed89c3024cc7315446"
Jul 27 21:27:23.476942 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:23.476906    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0493113b6f0a3c7be823456b6985de8e44e0c13e9105d9ed89c3024cc7315446"} err="failed to get container status \"0493113b6f0a3c7be823456b6985de8e44e0c13e9105d9ed89c3024cc7315446\": rpc error: code = NotFound desc = could not find container \"0493113b6f0a3c7be823456b6985de8e44e0c13e9105d9ed89c3024cc7315446\": container with ID starting with 0493113b6f0a3c7be823456b6985de8e44e0c13e9105d9ed89c3024cc7315446 not found: ID does not exist"
Jul 27 21:27:23.482050 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:23.482032    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"]
Jul 27 21:27:23.488298 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:23.488251    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0a095-predictor-745d9f4d88-hw9ff"]
Jul 27 21:27:24.571680 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:24.571646    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcde100a-1025-4ee1-a150-487a1086f4d3" path="/var/lib/kubelet/pods/dcde100a-1025-4ee1-a150-487a1086f4d3/volumes"
Jul 27 21:27:27.357010 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:27.356977    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7" podUID="e12f0e51-4d30-47ec-9730-7f1f552d3461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 27 21:27:27.460493 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:27.460462    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"
Jul 27 21:27:27.460913 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:27.460889    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq" podUID="eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 27 21:27:37.357157 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:37.357080    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"
Jul 27 21:27:37.461174 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:37.461130    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq" podUID="eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 27 21:27:47.461419 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:47.461384    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq" podUID="eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 27 21:27:49.931579 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:49.931545    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w"]
Jul 27 21:27:49.932082 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:49.931914    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcde100a-1025-4ee1-a150-487a1086f4d3" containerName="kube-rbac-proxy"
Jul 27 21:27:49.932082 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:49.931929    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcde100a-1025-4ee1-a150-487a1086f4d3" containerName="kube-rbac-proxy"
Jul 27 21:27:49.932082 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:49.931942    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcde100a-1025-4ee1-a150-487a1086f4d3" containerName="kserve-container"
Jul 27 21:27:49.932082 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:49.931950    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcde100a-1025-4ee1-a150-487a1086f4d3" containerName="kserve-container"
Jul 27 21:27:49.932082 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:49.932029    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcde100a-1025-4ee1-a150-487a1086f4d3" containerName="kserve-container"
Jul 27 21:27:49.932082 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:49.932040    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcde100a-1025-4ee1-a150-487a1086f4d3" containerName="kube-rbac-proxy"
Jul 27 21:27:49.936804 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:49.936785    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w"
Jul 27 21:27:49.939221 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:49.939197    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-e26c5-serving-cert\""
Jul 27 21:27:49.939619 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:49.939603    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-e26c5-kube-rbac-proxy-sar-config\""
Jul 27 21:27:49.944729 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:49.944706    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:49.950991 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:49.950947    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w"]
Jul 27 21:27:50.009509 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:50.009467    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb225b5-989f-44f8-907e-ef72ca380972-openshift-service-ca-bundle\") pod \"ensemble-graph-e26c5-5545679b85-cl99w\" (UID: \"8cb225b5-989f-44f8-907e-ef72ca380972\") " pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w"
Jul 27 21:27:50.009773 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:50.009517    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cb225b5-989f-44f8-907e-ef72ca380972-proxy-tls\") pod \"ensemble-graph-e26c5-5545679b85-cl99w\" (UID: \"8cb225b5-989f-44f8-907e-ef72ca380972\") " pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w"
Jul 27 21:27:50.110344 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:50.110314    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb225b5-989f-44f8-907e-ef72ca380972-openshift-service-ca-bundle\") pod \"ensemble-graph-e26c5-5545679b85-cl99w\" (UID: \"8cb225b5-989f-44f8-907e-ef72ca380972\") " pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w"
Jul 27 21:27:50.110508 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:50.110356    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cb225b5-989f-44f8-907e-ef72ca380972-proxy-tls\") pod \"ensemble-graph-e26c5-5545679b85-cl99w\" (UID: \"8cb225b5-989f-44f8-907e-ef72ca380972\") " pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w"
Jul 27 21:27:50.110937 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:50.110909    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb225b5-989f-44f8-907e-ef72ca380972-openshift-service-ca-bundle\") pod \"ensemble-graph-e26c5-5545679b85-cl99w\" (UID: \"8cb225b5-989f-44f8-907e-ef72ca380972\") " pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w"
Jul 27 21:27:50.112886 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:50.112857    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cb225b5-989f-44f8-907e-ef72ca380972-proxy-tls\") pod \"ensemble-graph-e26c5-5545679b85-cl99w\" (UID: \"8cb225b5-989f-44f8-907e-ef72ca380972\") " pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w"
Jul 27 21:27:50.142202 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:50.142168    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:50.247374 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:50.247287    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w"
Jul 27 21:27:50.373040 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:50.373010    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w"]
Jul 27 21:27:50.375939 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:27:50.375916    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb225b5_989f_44f8_907e_ef72ca380972.slice/crio-995d8f548db1f4f3ea3fb4f34227ccf1acfb1f7340877a8c2a3762ed87c054a9 WatchSource:0}: Error finding container 995d8f548db1f4f3ea3fb4f34227ccf1acfb1f7340877a8c2a3762ed87c054a9: Status 404 returned error can't find the container with id 995d8f548db1f4f3ea3fb4f34227ccf1acfb1f7340877a8c2a3762ed87c054a9
Jul 27 21:27:50.510423 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:50.510319    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:50.560762 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:50.560730    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w" event={"ID":"8cb225b5-989f-44f8-907e-ef72ca380972","Type":"ContainerStarted","Data":"b76f3ca67c2305ea0cc6842729b5d36b81a836d19a80e6a533e02618048e81bb"}
Jul 27 21:27:50.560941 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:50.560780    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w" event={"ID":"8cb225b5-989f-44f8-907e-ef72ca380972","Type":"ContainerStarted","Data":"995d8f548db1f4f3ea3fb4f34227ccf1acfb1f7340877a8c2a3762ed87c054a9"}
Jul 27 21:27:50.560941 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:50.560822    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w"
Jul 27 21:27:50.579943 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:50.579900    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w" podStartSLOduration=1.5798852110000001 podStartE2EDuration="1.579885211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:27:50.579245761 +0000 UTC m=+1450.514462165" watchObservedRunningTime="2026-07-27 21:27:50.579885211 +0000 UTC m=+1450.515101589"
Jul 27 21:27:54.138667 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:54.138622    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:56.572015 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:56.571990    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w"
Jul 27 21:27:57.461410 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:27:57.461372    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq" podUID="eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 27 21:28:07.462117 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:28:07.462088    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"
Jul 27 21:28:19.649692 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:28:19.649660    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:28:20.000933 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:28:20.000838    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:25.159164 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:29:25.159137    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:25.227493 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:29:25.227439    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:36:04.499361 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:04.499331    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w"]
Jul 27 21:36:04.499950 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:04.499573    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w" podUID="8cb225b5-989f-44f8-907e-ef72ca380972" containerName="ensemble-graph-e26c5" containerID="cri-o://b76f3ca67c2305ea0cc6842729b5d36b81a836d19a80e6a533e02618048e81bb" gracePeriod=30
Jul 27 21:36:04.568377 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:04.568339    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:36:04.776893 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:04.776865    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"]
Jul 27 21:36:04.780009 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:04.779991    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"
Jul 27 21:36:04.782343 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:04.782309    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-61561-kube-rbac-proxy-sar-config\""
Jul 27 21:36:04.782446 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:04.782365    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-61561-predictor-serving-cert\""
Jul 27 21:36:04.792810 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:04.792785    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"]
Jul 27 21:36:04.828843 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:04.828810    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"]
Jul 27 21:36:04.829112 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:04.829050    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7" podUID="e12f0e51-4d30-47ec-9730-7f1f552d3461" containerName="kserve-container" containerID="cri-o://29714127e0154cae514aba4a2a81937df30b97377a2b56a6b55c16d5f9b04efd" gracePeriod=30
Jul 27 21:36:04.829112 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:04.829106    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7" podUID="e12f0e51-4d30-47ec-9730-7f1f552d3461" containerName="kube-rbac-proxy" containerID="cri-o://175076e4ee812964cae16abc2828c5412e9de2917cf4e13ea885631ab67d441b" gracePeriod=30
Jul 27 21:36:04.867528 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:04.867498    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa05a7b-4022-4733-8d01-4e8a990dd69a-proxy-tls\") pod \"error-404-isvc-61561-predictor-545d79d4f6-h8m88\" (UID: \"8fa05a7b-4022-4733-8d01-4e8a990dd69a\") " pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"
Jul 27 21:36:04.867657 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:04.867561    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nls4h\" (UniqueName: \"kubernetes.io/projected/8fa05a7b-4022-4733-8d01-4e8a990dd69a-kube-api-access-nls4h\") pod \"error-404-isvc-61561-predictor-545d79d4f6-h8m88\" (UID: \"8fa05a7b-4022-4733-8d01-4e8a990dd69a\") " pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"
Jul 27 21:36:04.867657 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:04.867631    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fa05a7b-4022-4733-8d01-4e8a990dd69a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-61561-predictor-545d79d4f6-h8m88\" (UID: \"8fa05a7b-4022-4733-8d01-4e8a990dd69a\") " pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"
Jul 27 21:36:04.968501 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:04.968463    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fa05a7b-4022-4733-8d01-4e8a990dd69a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-61561-predictor-545d79d4f6-h8m88\" (UID: \"8fa05a7b-4022-4733-8d01-4e8a990dd69a\") " pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"
Jul 27 21:36:04.968672 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:04.968527    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa05a7b-4022-4733-8d01-4e8a990dd69a-proxy-tls\") pod \"error-404-isvc-61561-predictor-545d79d4f6-h8m88\" (UID: \"8fa05a7b-4022-4733-8d01-4e8a990dd69a\") " pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"
Jul 27 21:36:04.968672 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:04.968573    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nls4h\" (UniqueName: \"kubernetes.io/projected/8fa05a7b-4022-4733-8d01-4e8a990dd69a-kube-api-access-nls4h\") pod \"error-404-isvc-61561-predictor-545d79d4f6-h8m88\" (UID: \"8fa05a7b-4022-4733-8d01-4e8a990dd69a\") " pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"
Jul 27 21:36:04.968778 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:36:04.968707    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-61561-predictor-serving-cert: secret "error-404-isvc-61561-predictor-serving-cert" not found
Jul 27 21:36:04.969355 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:36:04.969267    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fa05a7b-4022-4733-8d01-4e8a990dd69a-proxy-tls podName:8fa05a7b-4022-4733-8d01-4e8a990dd69a nodeName:}" failed. No retries permitted until 2026-07-27 21:36:05.469228491 +0000 UTC m=+1945.404444849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8fa05a7b-4022-4733-8d01-4e8a990dd69a-proxy-tls") pod "error-404-isvc-61561-predictor-545d79d4f6-h8m88" (UID: "8fa05a7b-4022-4733-8d01-4e8a990dd69a") : secret "error-404-isvc-61561-predictor-serving-cert" not found
Jul 27 21:36:04.974845 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:04.969677    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fa05a7b-4022-4733-8d01-4e8a990dd69a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-61561-predictor-545d79d4f6-h8m88\" (UID: \"8fa05a7b-4022-4733-8d01-4e8a990dd69a\") " pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"
Jul 27 21:36:04.978396 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:04.978373    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nls4h\" (UniqueName: \"kubernetes.io/projected/8fa05a7b-4022-4733-8d01-4e8a990dd69a-kube-api-access-nls4h\") pod \"error-404-isvc-61561-predictor-545d79d4f6-h8m88\" (UID: \"8fa05a7b-4022-4733-8d01-4e8a990dd69a\") " pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"
Jul 27 21:36:05.236072 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:05.235990    2571 generic.go:356] "Generic (PLEG): container finished" podID="e12f0e51-4d30-47ec-9730-7f1f552d3461" containerID="175076e4ee812964cae16abc2828c5412e9de2917cf4e13ea885631ab67d441b" exitCode=2
Jul 27 21:36:05.236072 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:05.236029    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7" event={"ID":"e12f0e51-4d30-47ec-9730-7f1f552d3461","Type":"ContainerDied","Data":"175076e4ee812964cae16abc2828c5412e9de2917cf4e13ea885631ab67d441b"}
Jul 27 21:36:05.473088 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:05.473044    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa05a7b-4022-4733-8d01-4e8a990dd69a-proxy-tls\") pod \"error-404-isvc-61561-predictor-545d79d4f6-h8m88\" (UID: \"8fa05a7b-4022-4733-8d01-4e8a990dd69a\") " pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"
Jul 27 21:36:05.475560 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:05.475537    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa05a7b-4022-4733-8d01-4e8a990dd69a-proxy-tls\") pod \"error-404-isvc-61561-predictor-545d79d4f6-h8m88\" (UID: \"8fa05a7b-4022-4733-8d01-4e8a990dd69a\") " pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"
Jul 27 21:36:05.691091 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:05.691050    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"
Jul 27 21:36:05.819075 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:05.819043    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"]
Jul 27 21:36:05.822345 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:36:05.822320    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa05a7b_4022_4733_8d01_4e8a990dd69a.slice/crio-02d8635d69fbb6cebacd4cbccf30c14c4b49000653d29591b65bcf849204206c WatchSource:0}: Error finding container 02d8635d69fbb6cebacd4cbccf30c14c4b49000653d29591b65bcf849204206c: Status 404 returned error can't find the container with id 02d8635d69fbb6cebacd4cbccf30c14c4b49000653d29591b65bcf849204206c
Jul 27 21:36:06.229119 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:06.229035    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:36:06.241525 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:06.241489    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88" event={"ID":"8fa05a7b-4022-4733-8d01-4e8a990dd69a","Type":"ContainerStarted","Data":"86573d893cad676a80e57448156f8a19250a1bdd47e300bdb150aec340134ffd"}
Jul 27 21:36:06.241687 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:06.241552    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88" event={"ID":"8fa05a7b-4022-4733-8d01-4e8a990dd69a","Type":"ContainerStarted","Data":"f3861aecc5541ed5dab5974a2333dcf5ea9df0571a979ee126f5a6135e671de2"}
Jul 27 21:36:06.241687 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:06.241581    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88" event={"ID":"8fa05a7b-4022-4733-8d01-4e8a990dd69a","Type":"ContainerStarted","Data":"02d8635d69fbb6cebacd4cbccf30c14c4b49000653d29591b65bcf849204206c"}
Jul 27 21:36:06.241810 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:06.241718    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"
Jul 27 21:36:06.241810 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:06.241745    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"
Jul 27 21:36:06.243025 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:06.243000    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88" podUID="8fa05a7b-4022-4733-8d01-4e8a990dd69a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 27 21:36:06.260204 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:06.260160    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88" podStartSLOduration=2.260144666 podStartE2EDuration="2.260144666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:36:06.25842419 +0000 UTC m=+1946.193640571" watchObservedRunningTime="2026-07-27 21:36:06.260144666 +0000 UTC m=+1946.195361044"
Jul 27 21:36:06.568938 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:06.568905    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w" podUID="8cb225b5-989f-44f8-907e-ef72ca380972" containerName="ensemble-graph-e26c5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:36:07.245335 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:07.245298    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88" podUID="8fa05a7b-4022-4733-8d01-4e8a990dd69a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 27 21:36:07.352098 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:07.352062    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7" podUID="e12f0e51-4d30-47ec-9730-7f1f552d3461" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Jul 27 21:36:07.356916 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:07.356890    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7" podUID="e12f0e51-4d30-47ec-9730-7f1f552d3461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 27 21:36:08.080750 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.080727    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"
Jul 27 21:36:08.196145 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.196051    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vllgs\" (UniqueName: \"kubernetes.io/projected/e12f0e51-4d30-47ec-9730-7f1f552d3461-kube-api-access-vllgs\") pod \"e12f0e51-4d30-47ec-9730-7f1f552d3461\" (UID: \"e12f0e51-4d30-47ec-9730-7f1f552d3461\") "
Jul 27 21:36:08.196145 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.196128    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e12f0e51-4d30-47ec-9730-7f1f552d3461-kube-rbac-proxy-sar-config\") pod \"e12f0e51-4d30-47ec-9730-7f1f552d3461\" (UID: \"e12f0e51-4d30-47ec-9730-7f1f552d3461\") "
Jul 27 21:36:08.196372 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.196235    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e12f0e51-4d30-47ec-9730-7f1f552d3461-proxy-tls\") pod \"e12f0e51-4d30-47ec-9730-7f1f552d3461\" (UID: \"e12f0e51-4d30-47ec-9730-7f1f552d3461\") "
Jul 27 21:36:08.196451 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.196431    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12f0e51-4d30-47ec-9730-7f1f552d3461-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e12f0e51-4d30-47ec-9730-7f1f552d3461" (UID: "e12f0e51-4d30-47ec-9730-7f1f552d3461"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:36:08.198343 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.198323    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12f0e51-4d30-47ec-9730-7f1f552d3461-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e12f0e51-4d30-47ec-9730-7f1f552d3461" (UID: "e12f0e51-4d30-47ec-9730-7f1f552d3461"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:36:08.198445 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.198430    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12f0e51-4d30-47ec-9730-7f1f552d3461-kube-api-access-vllgs" (OuterVolumeSpecName: "kube-api-access-vllgs") pod "e12f0e51-4d30-47ec-9730-7f1f552d3461" (UID: "e12f0e51-4d30-47ec-9730-7f1f552d3461"). InnerVolumeSpecName "kube-api-access-vllgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:36:08.250174 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.250140    2571 generic.go:356] "Generic (PLEG): container finished" podID="e12f0e51-4d30-47ec-9730-7f1f552d3461" containerID="29714127e0154cae514aba4a2a81937df30b97377a2b56a6b55c16d5f9b04efd" exitCode=0
Jul 27 21:36:08.250620 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.250196    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"
Jul 27 21:36:08.250620 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.250212    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7" event={"ID":"e12f0e51-4d30-47ec-9730-7f1f552d3461","Type":"ContainerDied","Data":"29714127e0154cae514aba4a2a81937df30b97377a2b56a6b55c16d5f9b04efd"}
Jul 27 21:36:08.250620 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.250262    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7" event={"ID":"e12f0e51-4d30-47ec-9730-7f1f552d3461","Type":"ContainerDied","Data":"5e40d6f27f7421bf1cee4c2a1b5f8fe02b5e3bbfb7838929d7ad97253e3b4718"}
Jul 27 21:36:08.250620 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.250288    2571 scope.go:117] "RemoveContainer" containerID="175076e4ee812964cae16abc2828c5412e9de2917cf4e13ea885631ab67d441b"
Jul 27 21:36:08.261143 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.261126    2571 scope.go:117] "RemoveContainer" containerID="29714127e0154cae514aba4a2a81937df30b97377a2b56a6b55c16d5f9b04efd"
Jul 27 21:36:08.268336 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.268318    2571 scope.go:117] "RemoveContainer" containerID="175076e4ee812964cae16abc2828c5412e9de2917cf4e13ea885631ab67d441b"
Jul 27 21:36:08.268592 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:36:08.268575    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"175076e4ee812964cae16abc2828c5412e9de2917cf4e13ea885631ab67d441b\": container with ID starting with 175076e4ee812964cae16abc2828c5412e9de2917cf4e13ea885631ab67d441b not found: ID does not exist" containerID="175076e4ee812964cae16abc2828c5412e9de2917cf4e13ea885631ab67d441b"
Jul 27 21:36:08.268639 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.268601    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"175076e4ee812964cae16abc2828c5412e9de2917cf4e13ea885631ab67d441b"} err="failed to get container status \"175076e4ee812964cae16abc2828c5412e9de2917cf4e13ea885631ab67d441b\": rpc error: code = NotFound desc = could not find container \"175076e4ee812964cae16abc2828c5412e9de2917cf4e13ea885631ab67d441b\": container with ID starting with 175076e4ee812964cae16abc2828c5412e9de2917cf4e13ea885631ab67d441b not found: ID does not exist"
Jul 27 21:36:08.268639 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.268618    2571 scope.go:117] "RemoveContainer" containerID="29714127e0154cae514aba4a2a81937df30b97377a2b56a6b55c16d5f9b04efd"
Jul 27 21:36:08.268854 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:36:08.268837    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29714127e0154cae514aba4a2a81937df30b97377a2b56a6b55c16d5f9b04efd\": container with ID starting with 29714127e0154cae514aba4a2a81937df30b97377a2b56a6b55c16d5f9b04efd not found: ID does not exist" containerID="29714127e0154cae514aba4a2a81937df30b97377a2b56a6b55c16d5f9b04efd"
Jul 27 21:36:08.268903 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.268861    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29714127e0154cae514aba4a2a81937df30b97377a2b56a6b55c16d5f9b04efd"} err="failed to get container status \"29714127e0154cae514aba4a2a81937df30b97377a2b56a6b55c16d5f9b04efd\": rpc error: code = NotFound desc = could not find container \"29714127e0154cae514aba4a2a81937df30b97377a2b56a6b55c16d5f9b04efd\": container with ID starting with 29714127e0154cae514aba4a2a81937df30b97377a2b56a6b55c16d5f9b04efd not found: ID does not exist"
Jul 27 21:36:08.270677 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.270657    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"]
Jul 27 21:36:08.276912 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.276894    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e26c5-predictor-7897bcc8dd-zplm7"]
Jul 27 21:36:08.297520 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.297495    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e12f0e51-4d30-47ec-9730-7f1f552d3461-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:36:08.297520 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.297517    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e12f0e51-4d30-47ec-9730-7f1f552d3461-proxy-tls\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:36:08.297650 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.297528    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vllgs\" (UniqueName: \"kubernetes.io/projected/e12f0e51-4d30-47ec-9730-7f1f552d3461-kube-api-access-vllgs\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:36:08.572854 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:08.572823    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e12f0e51-4d30-47ec-9730-7f1f552d3461" path="/var/lib/kubelet/pods/e12f0e51-4d30-47ec-9730-7f1f552d3461/volumes"
Jul 27 21:36:11.573147 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:11.573116    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w" podUID="8cb225b5-989f-44f8-907e-ef72ca380972" containerName="ensemble-graph-e26c5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:36:12.250068 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:12.250041    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"
Jul 27 21:36:12.250497 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:12.250471    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88" podUID="8fa05a7b-4022-4733-8d01-4e8a990dd69a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 27 21:36:16.568456 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:16.568419    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w" podUID="8cb225b5-989f-44f8-907e-ef72ca380972" containerName="ensemble-graph-e26c5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:36:16.572012 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:16.571990    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w"
Jul 27 21:36:21.568692 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:21.568646    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w" podUID="8cb225b5-989f-44f8-907e-ef72ca380972" containerName="ensemble-graph-e26c5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:36:22.251363 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:22.251324    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88" podUID="8fa05a7b-4022-4733-8d01-4e8a990dd69a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 27 21:36:26.568830 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:26.568791    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w" podUID="8cb225b5-989f-44f8-907e-ef72ca380972" containerName="ensemble-graph-e26c5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:36:31.568714 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:31.568673    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w" podUID="8cb225b5-989f-44f8-907e-ef72ca380972" containerName="ensemble-graph-e26c5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:36:32.250982 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:32.250929    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88" podUID="8fa05a7b-4022-4733-8d01-4e8a990dd69a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 27 21:36:34.305400 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.305368    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:36:34.365600 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.365571    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"]
Jul 27 21:36:34.365867 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.365842    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq" podUID="eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d" containerName="kserve-container" containerID="cri-o://074504426d783abccaa8b33b1c27578b6cdb87e79c37a84cd89ccca3b323923b" gracePeriod=30
Jul 27 21:36:34.365987 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.365927    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq" podUID="eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d" containerName="kube-rbac-proxy" containerID="cri-o://1af79d0721623fe8dce313488d1a82cfad38f88ac757f46f4ceb85749330cc43" gracePeriod=30
Jul 27 21:36:34.395214 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.395185    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:36:34.465257 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.465224    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:36:34.578557 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.578507    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"]
Jul 27 21:36:34.578915 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.578895    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e12f0e51-4d30-47ec-9730-7f1f552d3461" containerName="kube-rbac-proxy"
Jul 27 21:36:34.578915 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.578914    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e12f0e51-4d30-47ec-9730-7f1f552d3461" containerName="kube-rbac-proxy"
Jul 27 21:36:34.579462 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.578944    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e12f0e51-4d30-47ec-9730-7f1f552d3461" containerName="kserve-container"
Jul 27 21:36:34.579462 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.578971    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e12f0e51-4d30-47ec-9730-7f1f552d3461" containerName="kserve-container"
Jul 27 21:36:34.579462 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.579067    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e12f0e51-4d30-47ec-9730-7f1f552d3461" containerName="kserve-container"
Jul 27 21:36:34.579462 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.579079    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e12f0e51-4d30-47ec-9730-7f1f552d3461" containerName="kube-rbac-proxy"
Jul 27 21:36:34.587547 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.586172    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"
Jul 27 21:36:34.589138 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.588383    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:36:34.589138 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.588801    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-beb6a-predictor-serving-cert\""
Jul 27 21:36:34.590119 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.590100    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-beb6a-kube-rbac-proxy-sar-config\""
Jul 27 21:36:34.594551 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.594531    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"]
Jul 27 21:36:34.659619 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.659597    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w"
Jul 27 21:36:34.674281 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.674249    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:36:34.686794 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.686772    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb225b5-989f-44f8-907e-ef72ca380972-openshift-service-ca-bundle\") pod \"8cb225b5-989f-44f8-907e-ef72ca380972\" (UID: \"8cb225b5-989f-44f8-907e-ef72ca380972\") "
Jul 27 21:36:34.686862 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.686810    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cb225b5-989f-44f8-907e-ef72ca380972-proxy-tls\") pod \"8cb225b5-989f-44f8-907e-ef72ca380972\" (UID: \"8cb225b5-989f-44f8-907e-ef72ca380972\") "
Jul 27 21:36:34.687003 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.686985    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bvt5\" (UniqueName: \"kubernetes.io/projected/240e7788-4b88-4b27-984f-0a35b90ceaa3-kube-api-access-6bvt5\") pod \"error-404-isvc-beb6a-predictor-7c99f78746-rr2sm\" (UID: \"240e7788-4b88-4b27-984f-0a35b90ceaa3\") " pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"
Jul 27 21:36:34.687066 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.687048    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/240e7788-4b88-4b27-984f-0a35b90ceaa3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-beb6a-predictor-7c99f78746-rr2sm\" (UID: \"240e7788-4b88-4b27-984f-0a35b90ceaa3\") " pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"
Jul 27 21:36:34.687104 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.687088    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/240e7788-4b88-4b27-984f-0a35b90ceaa3-proxy-tls\") pod \"error-404-isvc-beb6a-predictor-7c99f78746-rr2sm\" (UID: \"240e7788-4b88-4b27-984f-0a35b90ceaa3\") " pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"
Jul 27 21:36:34.687177 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.687159    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb225b5-989f-44f8-907e-ef72ca380972-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "8cb225b5-989f-44f8-907e-ef72ca380972" (UID: "8cb225b5-989f-44f8-907e-ef72ca380972"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:36:34.689273 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.689237    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb225b5-989f-44f8-907e-ef72ca380972-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8cb225b5-989f-44f8-907e-ef72ca380972" (UID: "8cb225b5-989f-44f8-907e-ef72ca380972"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:36:34.788269 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.788239    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/240e7788-4b88-4b27-984f-0a35b90ceaa3-proxy-tls\") pod \"error-404-isvc-beb6a-predictor-7c99f78746-rr2sm\" (UID: \"240e7788-4b88-4b27-984f-0a35b90ceaa3\") " pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"
Jul 27 21:36:34.788437 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.788296    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bvt5\" (UniqueName: \"kubernetes.io/projected/240e7788-4b88-4b27-984f-0a35b90ceaa3-kube-api-access-6bvt5\") pod \"error-404-isvc-beb6a-predictor-7c99f78746-rr2sm\" (UID: \"240e7788-4b88-4b27-984f-0a35b90ceaa3\") " pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"
Jul 27 21:36:34.788437 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.788331    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/240e7788-4b88-4b27-984f-0a35b90ceaa3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-beb6a-predictor-7c99f78746-rr2sm\" (UID: \"240e7788-4b88-4b27-984f-0a35b90ceaa3\") " pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"
Jul 27 21:36:34.788437 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.788363    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb225b5-989f-44f8-907e-ef72ca380972-openshift-service-ca-bundle\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:36:34.788437 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.788373    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cb225b5-989f-44f8-907e-ef72ca380972-proxy-tls\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:36:34.789010 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.788990    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/240e7788-4b88-4b27-984f-0a35b90ceaa3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-beb6a-predictor-7c99f78746-rr2sm\" (UID: \"240e7788-4b88-4b27-984f-0a35b90ceaa3\") " pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"
Jul 27 21:36:34.790872 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.790850    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/240e7788-4b88-4b27-984f-0a35b90ceaa3-proxy-tls\") pod \"error-404-isvc-beb6a-predictor-7c99f78746-rr2sm\" (UID: \"240e7788-4b88-4b27-984f-0a35b90ceaa3\") " pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"
Jul 27 21:36:34.797458 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.797435    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bvt5\" (UniqueName: \"kubernetes.io/projected/240e7788-4b88-4b27-984f-0a35b90ceaa3-kube-api-access-6bvt5\") pod \"error-404-isvc-beb6a-predictor-7c99f78746-rr2sm\" (UID: \"240e7788-4b88-4b27-984f-0a35b90ceaa3\") " pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"
Jul 27 21:36:34.901464 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:34.901386    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"
Jul 27 21:36:35.032275 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:35.032246    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"]
Jul 27 21:36:35.035484 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:36:35.035458    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240e7788_4b88_4b27_984f_0a35b90ceaa3.slice/crio-2a8a847dcce43def3d29f071aeb1669921e90f461a4d7ea0be000c3436079c16 WatchSource:0}: Error finding container 2a8a847dcce43def3d29f071aeb1669921e90f461a4d7ea0be000c3436079c16: Status 404 returned error can't find the container with id 2a8a847dcce43def3d29f071aeb1669921e90f461a4d7ea0be000c3436079c16
Jul 27 21:36:35.345366 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:35.345332    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm" event={"ID":"240e7788-4b88-4b27-984f-0a35b90ceaa3","Type":"ContainerStarted","Data":"d98087b257cd28fc129118ddd7471302abd689f588d3df91a489bfae9c79e70d"}
Jul 27 21:36:35.345889 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:35.345393    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm" event={"ID":"240e7788-4b88-4b27-984f-0a35b90ceaa3","Type":"ContainerStarted","Data":"455a16382717400dc47b20379bdaed21a66a651da434dd2150e3cce5cf3127b5"}
Jul 27 21:36:35.345889 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:35.345424    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"
Jul 27 21:36:35.345889 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:35.345444    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm" event={"ID":"240e7788-4b88-4b27-984f-0a35b90ceaa3","Type":"ContainerStarted","Data":"2a8a847dcce43def3d29f071aeb1669921e90f461a4d7ea0be000c3436079c16"}
Jul 27 21:36:35.346602 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:35.346582    2571 generic.go:356] "Generic (PLEG): container finished" podID="8cb225b5-989f-44f8-907e-ef72ca380972" containerID="b76f3ca67c2305ea0cc6842729b5d36b81a836d19a80e6a533e02618048e81bb" exitCode=0
Jul 27 21:36:35.346664 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:35.346622    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w"
Jul 27 21:36:35.346664 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:35.346649    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w" event={"ID":"8cb225b5-989f-44f8-907e-ef72ca380972","Type":"ContainerDied","Data":"b76f3ca67c2305ea0cc6842729b5d36b81a836d19a80e6a533e02618048e81bb"}
Jul 27 21:36:35.346727 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:35.346681    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w" event={"ID":"8cb225b5-989f-44f8-907e-ef72ca380972","Type":"ContainerDied","Data":"995d8f548db1f4f3ea3fb4f34227ccf1acfb1f7340877a8c2a3762ed87c054a9"}
Jul 27 21:36:35.346727 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:35.346702    2571 scope.go:117] "RemoveContainer" containerID="b76f3ca67c2305ea0cc6842729b5d36b81a836d19a80e6a533e02618048e81bb"
Jul 27 21:36:35.348705 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:35.348679    2571 generic.go:356] "Generic (PLEG): container finished" podID="eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d" containerID="1af79d0721623fe8dce313488d1a82cfad38f88ac757f46f4ceb85749330cc43" exitCode=2
Jul 27 21:36:35.348826 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:35.348749    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq" event={"ID":"eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d","Type":"ContainerDied","Data":"1af79d0721623fe8dce313488d1a82cfad38f88ac757f46f4ceb85749330cc43"}
Jul 27 21:36:35.355674 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:35.355657    2571 scope.go:117] "RemoveContainer" containerID="b76f3ca67c2305ea0cc6842729b5d36b81a836d19a80e6a533e02618048e81bb"
Jul 27 21:36:35.355908 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:36:35.355891    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b76f3ca67c2305ea0cc6842729b5d36b81a836d19a80e6a533e02618048e81bb\": container with ID starting with b76f3ca67c2305ea0cc6842729b5d36b81a836d19a80e6a533e02618048e81bb not found: ID does not exist" containerID="b76f3ca67c2305ea0cc6842729b5d36b81a836d19a80e6a533e02618048e81bb"
Jul 27 21:36:35.355986 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:35.355917    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b76f3ca67c2305ea0cc6842729b5d36b81a836d19a80e6a533e02618048e81bb"} err="failed to get container status \"b76f3ca67c2305ea0cc6842729b5d36b81a836d19a80e6a533e02618048e81bb\": rpc error: code = NotFound desc = could not find container \"b76f3ca67c2305ea0cc6842729b5d36b81a836d19a80e6a533e02618048e81bb\": container with ID starting with b76f3ca67c2305ea0cc6842729b5d36b81a836d19a80e6a533e02618048e81bb not found: ID does not exist"
Jul 27 21:36:35.363094 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:35.363057    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm" podStartSLOduration=1.363043673 podStartE2EDuration="1.363043673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:36:35.361745445 +0000 UTC m=+1975.296961825" watchObservedRunningTime="2026-07-27 21:36:35.363043673 +0000 UTC m=+1975.298260052"
Jul 27 21:36:35.372148 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:35.372128    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w"]
Jul 27 21:36:35.377429 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:35.377407    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e26c5-5545679b85-cl99w"]
Jul 27 21:36:35.397207 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:35.397176    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:36:36.352855 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:36.352825    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"
Jul 27 21:36:36.354102 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:36.354078    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm" podUID="240e7788-4b88-4b27-984f-0a35b90ceaa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 27 21:36:36.572061 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:36.572029    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cb225b5-989f-44f8-907e-ef72ca380972" path="/var/lib/kubelet/pods/8cb225b5-989f-44f8-907e-ef72ca380972/volumes"
Jul 27 21:36:37.356504 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:37.356463    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm" podUID="240e7788-4b88-4b27-984f-0a35b90ceaa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 27 21:36:37.456360 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:37.456323    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq" podUID="eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 27 21:36:37.461617 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:37.461593    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq" podUID="eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 27 21:36:37.817798 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:37.817773    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"
Jul 27 21:36:37.826028 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:37.826007    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:36:37.918623 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:37.918592    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d-kube-rbac-proxy-sar-config\") pod \"eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d\" (UID: \"eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d\") "
Jul 27 21:36:37.918788 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:37.918641    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d-proxy-tls\") pod \"eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d\" (UID: \"eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d\") "
Jul 27 21:36:37.918788 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:37.918709    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwrtt\" (UniqueName: \"kubernetes.io/projected/eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d-kube-api-access-jwrtt\") pod \"eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d\" (UID: \"eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d\") "
Jul 27 21:36:37.918985 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:37.918933    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d" (UID: "eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:36:37.920780 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:37.920759    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d" (UID: "eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:36:37.920863 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:37.920839    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d-kube-api-access-jwrtt" (OuterVolumeSpecName: "kube-api-access-jwrtt") pod "eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d" (UID: "eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d"). InnerVolumeSpecName "kube-api-access-jwrtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:36:38.020066 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:38.020036    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwrtt\" (UniqueName: \"kubernetes.io/projected/eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d-kube-api-access-jwrtt\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:36:38.020066 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:38.020064    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:36:38.020235 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:38.020074    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d-proxy-tls\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:36:38.361742 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:38.361651    2571 generic.go:356] "Generic (PLEG): container finished" podID="eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d" containerID="074504426d783abccaa8b33b1c27578b6cdb87e79c37a84cd89ccca3b323923b" exitCode=0
Jul 27 21:36:38.361742 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:38.361709    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"
Jul 27 21:36:38.361742 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:38.361709    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq" event={"ID":"eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d","Type":"ContainerDied","Data":"074504426d783abccaa8b33b1c27578b6cdb87e79c37a84cd89ccca3b323923b"}
Jul 27 21:36:38.362275 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:38.361771    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq" event={"ID":"eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d","Type":"ContainerDied","Data":"c14368f44f81fa3c7025743fe683ee325881c7cb7e79cafdf6625551bfae3e70"}
Jul 27 21:36:38.362275 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:38.361799    2571 scope.go:117] "RemoveContainer" containerID="1af79d0721623fe8dce313488d1a82cfad38f88ac757f46f4ceb85749330cc43"
Jul 27 21:36:38.370389 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:38.370371    2571 scope.go:117] "RemoveContainer" containerID="074504426d783abccaa8b33b1c27578b6cdb87e79c37a84cd89ccca3b323923b"
Jul 27 21:36:38.377632 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:38.377614    2571 scope.go:117] "RemoveContainer" containerID="1af79d0721623fe8dce313488d1a82cfad38f88ac757f46f4ceb85749330cc43"
Jul 27 21:36:38.377856 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:36:38.377838    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af79d0721623fe8dce313488d1a82cfad38f88ac757f46f4ceb85749330cc43\": container with ID starting with 1af79d0721623fe8dce313488d1a82cfad38f88ac757f46f4ceb85749330cc43 not found: ID does not exist" containerID="1af79d0721623fe8dce313488d1a82cfad38f88ac757f46f4ceb85749330cc43"
Jul 27 21:36:38.377917 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:38.377863    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af79d0721623fe8dce313488d1a82cfad38f88ac757f46f4ceb85749330cc43"} err="failed to get container status \"1af79d0721623fe8dce313488d1a82cfad38f88ac757f46f4ceb85749330cc43\": rpc error: code = NotFound desc = could not find container \"1af79d0721623fe8dce313488d1a82cfad38f88ac757f46f4ceb85749330cc43\": container with ID starting with 1af79d0721623fe8dce313488d1a82cfad38f88ac757f46f4ceb85749330cc43 not found: ID does not exist"
Jul 27 21:36:38.377917 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:38.377878    2571 scope.go:117] "RemoveContainer" containerID="074504426d783abccaa8b33b1c27578b6cdb87e79c37a84cd89ccca3b323923b"
Jul 27 21:36:38.378122 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:36:38.378106    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"074504426d783abccaa8b33b1c27578b6cdb87e79c37a84cd89ccca3b323923b\": container with ID starting with 074504426d783abccaa8b33b1c27578b6cdb87e79c37a84cd89ccca3b323923b not found: ID does not exist" containerID="074504426d783abccaa8b33b1c27578b6cdb87e79c37a84cd89ccca3b323923b"
Jul 27 21:36:38.378165 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:38.378126    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"074504426d783abccaa8b33b1c27578b6cdb87e79c37a84cd89ccca3b323923b"} err="failed to get container status \"074504426d783abccaa8b33b1c27578b6cdb87e79c37a84cd89ccca3b323923b\": rpc error: code = NotFound desc = could not find container \"074504426d783abccaa8b33b1c27578b6cdb87e79c37a84cd89ccca3b323923b\": container with ID starting with 074504426d783abccaa8b33b1c27578b6cdb87e79c37a84cd89ccca3b323923b not found: ID does not exist"
Jul 27 21:36:38.382197 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:38.382177    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"]
Jul 27 21:36:38.387346 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:38.387326    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0f5d7-predictor-655fb59d6-xv9mq"]
Jul 27 21:36:38.572298 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:38.572269    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d" path="/var/lib/kubelet/pods/eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d/volumes"
Jul 27 21:36:42.251172 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:42.251137    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88" podUID="8fa05a7b-4022-4733-8d01-4e8a990dd69a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 27 21:36:42.360861 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:42.360835    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"
Jul 27 21:36:42.361298 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:42.361270    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm" podUID="240e7788-4b88-4b27-984f-0a35b90ceaa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 27 21:36:48.708728 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:48.708697    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:36:48.765437 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:48.765411    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:36:52.251123 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:52.251096    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"
Jul 27 21:36:52.361893 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:36:52.361858    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm" podUID="240e7788-4b88-4b27-984f-0a35b90ceaa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 27 21:37:02.361305 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:02.361265    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm" podUID="240e7788-4b88-4b27-984f-0a35b90ceaa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 27 21:37:04.785047 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:04.785016    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:04.887925 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:04.887894    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:12.362192 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:12.362151    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm" podUID="240e7788-4b88-4b27-984f-0a35b90ceaa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 27 21:37:14.871771 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:14.871734    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:15.001008 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.000971    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:15.026466 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.026434    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"]
Jul 27 21:37:15.026705 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.026679    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88" podUID="8fa05a7b-4022-4733-8d01-4e8a990dd69a" containerName="kserve-container" containerID="cri-o://f3861aecc5541ed5dab5974a2333dcf5ea9df0571a979ee126f5a6135e671de2" gracePeriod=30
Jul 27 21:37:15.026773 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.026699    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88" podUID="8fa05a7b-4022-4733-8d01-4e8a990dd69a" containerName="kube-rbac-proxy" containerID="cri-o://86573d893cad676a80e57448156f8a19250a1bdd47e300bdb150aec340134ffd" gracePeriod=30
Jul 27 21:37:15.095484 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.095444    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:15.130266 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.130183    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"]
Jul 27 21:37:15.130617 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.130590    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d" containerName="kserve-container"
Jul 27 21:37:15.130617 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.130610    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d" containerName="kserve-container"
Jul 27 21:37:15.130617 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.130620    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d" containerName="kube-rbac-proxy"
Jul 27 21:37:15.130617 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.130626    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d" containerName="kube-rbac-proxy"
Jul 27 21:37:15.130897 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.130633    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cb225b5-989f-44f8-907e-ef72ca380972" containerName="ensemble-graph-e26c5"
Jul 27 21:37:15.130897 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.130638    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cb225b5-989f-44f8-907e-ef72ca380972" containerName="ensemble-graph-e26c5"
Jul 27 21:37:15.130897 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.130687    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d" containerName="kserve-container"
Jul 27 21:37:15.130897 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.130695    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba7c1dc-88e9-4f1a-83ea-adb3be7f8d6d" containerName="kube-rbac-proxy"
Jul 27 21:37:15.130897 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.130701    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cb225b5-989f-44f8-907e-ef72ca380972" containerName="ensemble-graph-e26c5"
Jul 27 21:37:15.133716 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.133700    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"
Jul 27 21:37:15.136186 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.136160    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b2e96-kube-rbac-proxy-sar-config\""
Jul 27 21:37:15.136358 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.136340    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b2e96-predictor-serving-cert\""
Jul 27 21:37:15.144417 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.144387    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"]
Jul 27 21:37:15.194408 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.194372    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:15.294229 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.294191    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:15.311182 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.311145    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p7vw\" (UniqueName: \"kubernetes.io/projected/8df6bafa-cbe8-407a-a34c-e336c29ae291-kube-api-access-7p7vw\") pod \"error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn\" (UID: \"8df6bafa-cbe8-407a-a34c-e336c29ae291\") " pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"
Jul 27 21:37:15.311343 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.311191    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8df6bafa-cbe8-407a-a34c-e336c29ae291-proxy-tls\") pod \"error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn\" (UID: \"8df6bafa-cbe8-407a-a34c-e336c29ae291\") " pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"
Jul 27 21:37:15.311343 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.311262    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8df6bafa-cbe8-407a-a34c-e336c29ae291-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn\" (UID: \"8df6bafa-cbe8-407a-a34c-e336c29ae291\") " pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"
Jul 27 21:37:15.394145 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.394047    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:15.412612 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.412579    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p7vw\" (UniqueName: \"kubernetes.io/projected/8df6bafa-cbe8-407a-a34c-e336c29ae291-kube-api-access-7p7vw\") pod \"error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn\" (UID: \"8df6bafa-cbe8-407a-a34c-e336c29ae291\") " pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"
Jul 27 21:37:15.412775 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.412619    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8df6bafa-cbe8-407a-a34c-e336c29ae291-proxy-tls\") pod \"error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn\" (UID: \"8df6bafa-cbe8-407a-a34c-e336c29ae291\") " pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"
Jul 27 21:37:15.412775 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.412652    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8df6bafa-cbe8-407a-a34c-e336c29ae291-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn\" (UID: \"8df6bafa-cbe8-407a-a34c-e336c29ae291\") " pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"
Jul 27 21:37:15.413341 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.413323    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8df6bafa-cbe8-407a-a34c-e336c29ae291-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn\" (UID: \"8df6bafa-cbe8-407a-a34c-e336c29ae291\") " pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"
Jul 27 21:37:15.415253 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.415235    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8df6bafa-cbe8-407a-a34c-e336c29ae291-proxy-tls\") pod \"error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn\" (UID: \"8df6bafa-cbe8-407a-a34c-e336c29ae291\") " pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"
Jul 27 21:37:15.420989 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.420944    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p7vw\" (UniqueName: \"kubernetes.io/projected/8df6bafa-cbe8-407a-a34c-e336c29ae291-kube-api-access-7p7vw\") pod \"error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn\" (UID: \"8df6bafa-cbe8-407a-a34c-e336c29ae291\") " pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"
Jul 27 21:37:15.445854 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.445829    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"
Jul 27 21:37:15.492141 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.492108    2571 generic.go:356] "Generic (PLEG): container finished" podID="8fa05a7b-4022-4733-8d01-4e8a990dd69a" containerID="86573d893cad676a80e57448156f8a19250a1bdd47e300bdb150aec340134ffd" exitCode=2
Jul 27 21:37:15.492298 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.492162    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88" event={"ID":"8fa05a7b-4022-4733-8d01-4e8a990dd69a","Type":"ContainerDied","Data":"86573d893cad676a80e57448156f8a19250a1bdd47e300bdb150aec340134ffd"}
Jul 27 21:37:15.582271 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:15.582242    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"]
Jul 27 21:37:15.585094 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:37:15.585068    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df6bafa_cbe8_407a_a34c_e336c29ae291.slice/crio-38b9f08863faaa9bc3e859292d560feb25b4a36e7639be4b035e30d3801739ef WatchSource:0}: Error finding container 38b9f08863faaa9bc3e859292d560feb25b4a36e7639be4b035e30d3801739ef: Status 404 returned error can't find the container with id 38b9f08863faaa9bc3e859292d560feb25b4a36e7639be4b035e30d3801739ef
Jul 27 21:37:16.498663 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:16.498625    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn" event={"ID":"8df6bafa-cbe8-407a-a34c-e336c29ae291","Type":"ContainerStarted","Data":"e2415acbbfe6f580d0e4e9d06c5f01ea4de8e33ef99005133c4e802db6badd97"}
Jul 27 21:37:16.499151 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:16.498680    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn" event={"ID":"8df6bafa-cbe8-407a-a34c-e336c29ae291","Type":"ContainerStarted","Data":"d234f18860391e97b06f16b909900c62d2fcd3764ceef061e87d11cc50dd60e1"}
Jul 27 21:37:16.499151 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:16.498707    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn" event={"ID":"8df6bafa-cbe8-407a-a34c-e336c29ae291","Type":"ContainerStarted","Data":"38b9f08863faaa9bc3e859292d560feb25b4a36e7639be4b035e30d3801739ef"}
Jul 27 21:37:16.499151 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:16.498784    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"
Jul 27 21:37:16.517435 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:16.517377    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn" podStartSLOduration=1.517360577 podStartE2EDuration="1.517360577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:37:16.516219794 +0000 UTC m=+2016.451436175" watchObservedRunningTime="2026-07-27 21:37:16.517360577 +0000 UTC m=+2016.452576955"
Jul 27 21:37:17.245862 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:17.245821    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88" podUID="8fa05a7b-4022-4733-8d01-4e8a990dd69a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 27 21:37:17.502593 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:17.502514    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"
Jul 27 21:37:17.503757 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:17.503731    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn" podUID="8df6bafa-cbe8-407a-a34c-e336c29ae291" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 27 21:37:18.274221 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.274197    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"
Jul 27 21:37:18.339111 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.339076    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa05a7b-4022-4733-8d01-4e8a990dd69a-proxy-tls\") pod \"8fa05a7b-4022-4733-8d01-4e8a990dd69a\" (UID: \"8fa05a7b-4022-4733-8d01-4e8a990dd69a\") "
Jul 27 21:37:18.339282 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.339133    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nls4h\" (UniqueName: \"kubernetes.io/projected/8fa05a7b-4022-4733-8d01-4e8a990dd69a-kube-api-access-nls4h\") pod \"8fa05a7b-4022-4733-8d01-4e8a990dd69a\" (UID: \"8fa05a7b-4022-4733-8d01-4e8a990dd69a\") "
Jul 27 21:37:18.339282 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.339165    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fa05a7b-4022-4733-8d01-4e8a990dd69a-kube-rbac-proxy-sar-config\") pod \"8fa05a7b-4022-4733-8d01-4e8a990dd69a\" (UID: \"8fa05a7b-4022-4733-8d01-4e8a990dd69a\") "
Jul 27 21:37:18.339555 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.339523    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa05a7b-4022-4733-8d01-4e8a990dd69a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8fa05a7b-4022-4733-8d01-4e8a990dd69a" (UID: "8fa05a7b-4022-4733-8d01-4e8a990dd69a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:37:18.341381 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.341349    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa05a7b-4022-4733-8d01-4e8a990dd69a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8fa05a7b-4022-4733-8d01-4e8a990dd69a" (UID: "8fa05a7b-4022-4733-8d01-4e8a990dd69a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:37:18.341487 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.341463    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa05a7b-4022-4733-8d01-4e8a990dd69a-kube-api-access-nls4h" (OuterVolumeSpecName: "kube-api-access-nls4h") pod "8fa05a7b-4022-4733-8d01-4e8a990dd69a" (UID: "8fa05a7b-4022-4733-8d01-4e8a990dd69a"). InnerVolumeSpecName "kube-api-access-nls4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:37:18.439704 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.439656    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa05a7b-4022-4733-8d01-4e8a990dd69a-proxy-tls\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:37:18.439704 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.439698    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nls4h\" (UniqueName: \"kubernetes.io/projected/8fa05a7b-4022-4733-8d01-4e8a990dd69a-kube-api-access-nls4h\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:37:18.439704 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.439711    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fa05a7b-4022-4733-8d01-4e8a990dd69a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:37:18.507511 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.507477    2571 generic.go:356] "Generic (PLEG): container finished" podID="8fa05a7b-4022-4733-8d01-4e8a990dd69a" containerID="f3861aecc5541ed5dab5974a2333dcf5ea9df0571a979ee126f5a6135e671de2" exitCode=0
Jul 27 21:37:18.507945 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.507538    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88" event={"ID":"8fa05a7b-4022-4733-8d01-4e8a990dd69a","Type":"ContainerDied","Data":"f3861aecc5541ed5dab5974a2333dcf5ea9df0571a979ee126f5a6135e671de2"}
Jul 27 21:37:18.507945 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.507604    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88" event={"ID":"8fa05a7b-4022-4733-8d01-4e8a990dd69a","Type":"ContainerDied","Data":"02d8635d69fbb6cebacd4cbccf30c14c4b49000653d29591b65bcf849204206c"}
Jul 27 21:37:18.507945 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.507616    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"
Jul 27 21:37:18.507945 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.507649    2571 scope.go:117] "RemoveContainer" containerID="86573d893cad676a80e57448156f8a19250a1bdd47e300bdb150aec340134ffd"
Jul 27 21:37:18.507945 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.507778    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn" podUID="8df6bafa-cbe8-407a-a34c-e336c29ae291" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 27 21:37:18.516116 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.516100    2571 scope.go:117] "RemoveContainer" containerID="f3861aecc5541ed5dab5974a2333dcf5ea9df0571a979ee126f5a6135e671de2"
Jul 27 21:37:18.523361 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.523342    2571 scope.go:117] "RemoveContainer" containerID="86573d893cad676a80e57448156f8a19250a1bdd47e300bdb150aec340134ffd"
Jul 27 21:37:18.523631 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:37:18.523614    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86573d893cad676a80e57448156f8a19250a1bdd47e300bdb150aec340134ffd\": container with ID starting with 86573d893cad676a80e57448156f8a19250a1bdd47e300bdb150aec340134ffd not found: ID does not exist" containerID="86573d893cad676a80e57448156f8a19250a1bdd47e300bdb150aec340134ffd"
Jul 27 21:37:18.523680 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.523639    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86573d893cad676a80e57448156f8a19250a1bdd47e300bdb150aec340134ffd"} err="failed to get container status \"86573d893cad676a80e57448156f8a19250a1bdd47e300bdb150aec340134ffd\": rpc error: code = NotFound desc = could not find container \"86573d893cad676a80e57448156f8a19250a1bdd47e300bdb150aec340134ffd\": container with ID starting with 86573d893cad676a80e57448156f8a19250a1bdd47e300bdb150aec340134ffd not found: ID does not exist"
Jul 27 21:37:18.523680 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.523656    2571 scope.go:117] "RemoveContainer" containerID="f3861aecc5541ed5dab5974a2333dcf5ea9df0571a979ee126f5a6135e671de2"
Jul 27 21:37:18.523891 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:37:18.523874    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3861aecc5541ed5dab5974a2333dcf5ea9df0571a979ee126f5a6135e671de2\": container with ID starting with f3861aecc5541ed5dab5974a2333dcf5ea9df0571a979ee126f5a6135e671de2 not found: ID does not exist" containerID="f3861aecc5541ed5dab5974a2333dcf5ea9df0571a979ee126f5a6135e671de2"
Jul 27 21:37:18.523931 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.523898    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3861aecc5541ed5dab5974a2333dcf5ea9df0571a979ee126f5a6135e671de2"} err="failed to get container status \"f3861aecc5541ed5dab5974a2333dcf5ea9df0571a979ee126f5a6135e671de2\": rpc error: code = NotFound desc = could not find container \"f3861aecc5541ed5dab5974a2333dcf5ea9df0571a979ee126f5a6135e671de2\": container with ID starting with f3861aecc5541ed5dab5974a2333dcf5ea9df0571a979ee126f5a6135e671de2 not found: ID does not exist"
Jul 27 21:37:18.527421 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.527401    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"]
Jul 27 21:37:18.530577 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.530557    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-61561-predictor-545d79d4f6-h8m88"]
Jul 27 21:37:18.572217 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:18.572189    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fa05a7b-4022-4733-8d01-4e8a990dd69a" path="/var/lib/kubelet/pods/8fa05a7b-4022-4733-8d01-4e8a990dd69a/volumes"
Jul 27 21:37:22.362749 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:22.362714    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"
Jul 27 21:37:23.511859 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:23.511833    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"
Jul 27 21:37:23.512275 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:23.512158    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn" podUID="8df6bafa-cbe8-407a-a34c-e336c29ae291" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 27 21:37:33.512607 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:33.512564    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn" podUID="8df6bafa-cbe8-407a-a34c-e336c29ae291" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 27 21:37:34.522000 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:34.521944    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps"]
Jul 27 21:37:34.522354 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:34.522282    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa05a7b-4022-4733-8d01-4e8a990dd69a" containerName="kserve-container"
Jul 27 21:37:34.522354 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:34.522293    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa05a7b-4022-4733-8d01-4e8a990dd69a" containerName="kserve-container"
Jul 27 21:37:34.522354 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:34.522309    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa05a7b-4022-4733-8d01-4e8a990dd69a" containerName="kube-rbac-proxy"
Jul 27 21:37:34.522354 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:34.522314    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa05a7b-4022-4733-8d01-4e8a990dd69a" containerName="kube-rbac-proxy"
Jul 27 21:37:34.522479 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:34.522373    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fa05a7b-4022-4733-8d01-4e8a990dd69a" containerName="kube-rbac-proxy"
Jul 27 21:37:34.522479 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:34.522380    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fa05a7b-4022-4733-8d01-4e8a990dd69a" containerName="kserve-container"
Jul 27 21:37:34.526759 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:34.526743    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps"
Jul 27 21:37:34.528075 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:34.528049    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:34.529160 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:34.529144    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-beb6a-kube-rbac-proxy-sar-config\""
Jul 27 21:37:34.529246 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:34.529150    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-beb6a-serving-cert\""
Jul 27 21:37:34.534143 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:34.534123    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps"]
Jul 27 21:37:34.563659 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:34.563632    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42109737-e494-4909-8510-7683c70697c8-proxy-tls\") pod \"switch-graph-beb6a-78db6d8859-vxrps\" (UID: \"42109737-e494-4909-8510-7683c70697c8\") " pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps"
Jul 27 21:37:34.563758 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:34.563661    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42109737-e494-4909-8510-7683c70697c8-openshift-service-ca-bundle\") pod \"switch-graph-beb6a-78db6d8859-vxrps\" (UID: \"42109737-e494-4909-8510-7683c70697c8\") " pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps"
Jul 27 21:37:34.665076 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:34.665039    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42109737-e494-4909-8510-7683c70697c8-proxy-tls\") pod \"switch-graph-beb6a-78db6d8859-vxrps\" (UID: \"42109737-e494-4909-8510-7683c70697c8\") " pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps"
Jul 27 21:37:34.665288 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:34.665113    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42109737-e494-4909-8510-7683c70697c8-openshift-service-ca-bundle\") pod \"switch-graph-beb6a-78db6d8859-vxrps\" (UID: \"42109737-e494-4909-8510-7683c70697c8\") " pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps"
Jul 27 21:37:34.665894 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:34.665811    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42109737-e494-4909-8510-7683c70697c8-openshift-service-ca-bundle\") pod \"switch-graph-beb6a-78db6d8859-vxrps\" (UID: \"42109737-e494-4909-8510-7683c70697c8\") " pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps"
Jul 27 21:37:34.667547 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:34.667524    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42109737-e494-4909-8510-7683c70697c8-proxy-tls\") pod \"switch-graph-beb6a-78db6d8859-vxrps\" (UID: \"42109737-e494-4909-8510-7683c70697c8\") " pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps"
Jul 27 21:37:34.838163 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:34.838073    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps"
Jul 27 21:37:34.939637 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:34.939599    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:34.963584 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:34.963546    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps"]
Jul 27 21:37:34.966063 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:37:34.966034    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42109737_e494_4909_8510_7683c70697c8.slice/crio-63bad577293b406cc5fc9215e8451a66a95ec540a3581b12a2692e37a16acdfb WatchSource:0}: Error finding container 63bad577293b406cc5fc9215e8451a66a95ec540a3581b12a2692e37a16acdfb: Status 404 returned error can't find the container with id 63bad577293b406cc5fc9215e8451a66a95ec540a3581b12a2692e37a16acdfb
Jul 27 21:37:35.569012 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:35.568973    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps" event={"ID":"42109737-e494-4909-8510-7683c70697c8","Type":"ContainerStarted","Data":"2366360e552fcbda141ba2371a543d2a61a184f13a539e6742475bdf8e5c44bd"}
Jul 27 21:37:35.569391 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:35.569027    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps" event={"ID":"42109737-e494-4909-8510-7683c70697c8","Type":"ContainerStarted","Data":"63bad577293b406cc5fc9215e8451a66a95ec540a3581b12a2692e37a16acdfb"}
Jul 27 21:37:35.569391 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:35.569107    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps"
Jul 27 21:37:35.586467 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:35.586421    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps" podStartSLOduration=1.586408036 podStartE2EDuration="1.586408036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:37:35.584590269 +0000 UTC m=+2035.519806647" watchObservedRunningTime="2026-07-27 21:37:35.586408036 +0000 UTC m=+2035.521624415"
Jul 27 21:37:41.581581 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:41.581549    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps"
Jul 27 21:37:43.512941 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:43.512901    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn" podUID="8df6bafa-cbe8-407a-a34c-e336c29ae291" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 27 21:37:45.555917 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:45.555883    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:45.656104 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:45.656047    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:53.512633 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:37:53.512584    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn" podUID="8df6bafa-cbe8-407a-a34c-e336c29ae291" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 27 21:38:03.513594 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:38:03.513509    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"
Jul 27 21:38:09.213145 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:38:09.213108    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:38:15.101171 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:38:15.101131    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:38:15.219146 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:38:15.219113    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:27.041069 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:44:27.041040    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:45:27.227521 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:45:27.227488    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:46:29.703174 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:29.703128    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:46:29.827094 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:29.827067    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:46:29.848812 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:29.848783    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"]
Jul 27 21:46:29.849047 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:29.849016    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn" podUID="8df6bafa-cbe8-407a-a34c-e336c29ae291" containerName="kserve-container" containerID="cri-o://d234f18860391e97b06f16b909900c62d2fcd3764ceef061e87d11cc50dd60e1" gracePeriod=30
Jul 27 21:46:29.849191 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:29.849057    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn" podUID="8df6bafa-cbe8-407a-a34c-e336c29ae291" containerName="kube-rbac-proxy" containerID="cri-o://e2415acbbfe6f580d0e4e9d06c5f01ea4de8e33ef99005133c4e802db6badd97" gracePeriod=30
Jul 27 21:46:29.884774 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:29.884724    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:46:30.377280 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:30.377245    2571 generic.go:356] "Generic (PLEG): container finished" podID="8df6bafa-cbe8-407a-a34c-e336c29ae291" containerID="e2415acbbfe6f580d0e4e9d06c5f01ea4de8e33ef99005133c4e802db6badd97" exitCode=2
Jul 27 21:46:30.377448 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:30.377304    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn" event={"ID":"8df6bafa-cbe8-407a-a34c-e336c29ae291","Type":"ContainerDied","Data":"e2415acbbfe6f580d0e4e9d06c5f01ea4de8e33ef99005133c4e802db6badd97"}
Jul 27 21:46:32.695927 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:32.695905    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"
Jul 27 21:46:32.762217 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:32.762153    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8df6bafa-cbe8-407a-a34c-e336c29ae291-kube-rbac-proxy-sar-config\") pod \"8df6bafa-cbe8-407a-a34c-e336c29ae291\" (UID: \"8df6bafa-cbe8-407a-a34c-e336c29ae291\") "
Jul 27 21:46:32.762217 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:32.762198    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7p7vw\" (UniqueName: \"kubernetes.io/projected/8df6bafa-cbe8-407a-a34c-e336c29ae291-kube-api-access-7p7vw\") pod \"8df6bafa-cbe8-407a-a34c-e336c29ae291\" (UID: \"8df6bafa-cbe8-407a-a34c-e336c29ae291\") "
Jul 27 21:46:32.762217 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:32.762220    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8df6bafa-cbe8-407a-a34c-e336c29ae291-proxy-tls\") pod \"8df6bafa-cbe8-407a-a34c-e336c29ae291\" (UID: \"8df6bafa-cbe8-407a-a34c-e336c29ae291\") "
Jul 27 21:46:32.762478 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:32.762453    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8df6bafa-cbe8-407a-a34c-e336c29ae291-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8df6bafa-cbe8-407a-a34c-e336c29ae291" (UID: "8df6bafa-cbe8-407a-a34c-e336c29ae291"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:46:32.764340 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:32.764320    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df6bafa-cbe8-407a-a34c-e336c29ae291-kube-api-access-7p7vw" (OuterVolumeSpecName: "kube-api-access-7p7vw") pod "8df6bafa-cbe8-407a-a34c-e336c29ae291" (UID: "8df6bafa-cbe8-407a-a34c-e336c29ae291"). InnerVolumeSpecName "kube-api-access-7p7vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:46:32.764423 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:32.764336    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df6bafa-cbe8-407a-a34c-e336c29ae291-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8df6bafa-cbe8-407a-a34c-e336c29ae291" (UID: "8df6bafa-cbe8-407a-a34c-e336c29ae291"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:46:32.863394 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:32.863353    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p7vw\" (UniqueName: \"kubernetes.io/projected/8df6bafa-cbe8-407a-a34c-e336c29ae291-kube-api-access-7p7vw\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:46:32.863394 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:32.863389    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8df6bafa-cbe8-407a-a34c-e336c29ae291-proxy-tls\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:46:32.863394 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:32.863400    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8df6bafa-cbe8-407a-a34c-e336c29ae291-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:46:33.388906 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:33.388865    2571 generic.go:356] "Generic (PLEG): container finished" podID="8df6bafa-cbe8-407a-a34c-e336c29ae291" containerID="d234f18860391e97b06f16b909900c62d2fcd3764ceef061e87d11cc50dd60e1" exitCode=0
Jul 27 21:46:33.389113 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:33.388915    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn" event={"ID":"8df6bafa-cbe8-407a-a34c-e336c29ae291","Type":"ContainerDied","Data":"d234f18860391e97b06f16b909900c62d2fcd3764ceef061e87d11cc50dd60e1"}
Jul 27 21:46:33.389113 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:33.388934    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"
Jul 27 21:46:33.389113 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:33.389010    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn" event={"ID":"8df6bafa-cbe8-407a-a34c-e336c29ae291","Type":"ContainerDied","Data":"38b9f08863faaa9bc3e859292d560feb25b4a36e7639be4b035e30d3801739ef"}
Jul 27 21:46:33.389113 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:33.389042    2571 scope.go:117] "RemoveContainer" containerID="e2415acbbfe6f580d0e4e9d06c5f01ea4de8e33ef99005133c4e802db6badd97"
Jul 27 21:46:33.397868 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:33.397845    2571 scope.go:117] "RemoveContainer" containerID="d234f18860391e97b06f16b909900c62d2fcd3764ceef061e87d11cc50dd60e1"
Jul 27 21:46:33.404753 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:33.404733    2571 scope.go:117] "RemoveContainer" containerID="e2415acbbfe6f580d0e4e9d06c5f01ea4de8e33ef99005133c4e802db6badd97"
Jul 27 21:46:33.405015 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:46:33.404999    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2415acbbfe6f580d0e4e9d06c5f01ea4de8e33ef99005133c4e802db6badd97\": container with ID starting with e2415acbbfe6f580d0e4e9d06c5f01ea4de8e33ef99005133c4e802db6badd97 not found: ID does not exist" containerID="e2415acbbfe6f580d0e4e9d06c5f01ea4de8e33ef99005133c4e802db6badd97"
Jul 27 21:46:33.405068 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:33.405023    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2415acbbfe6f580d0e4e9d06c5f01ea4de8e33ef99005133c4e802db6badd97"} err="failed to get container status \"e2415acbbfe6f580d0e4e9d06c5f01ea4de8e33ef99005133c4e802db6badd97\": rpc error: code = NotFound desc = could not find container \"e2415acbbfe6f580d0e4e9d06c5f01ea4de8e33ef99005133c4e802db6badd97\": container with ID starting with e2415acbbfe6f580d0e4e9d06c5f01ea4de8e33ef99005133c4e802db6badd97 not found: ID does not exist"
Jul 27 21:46:33.405068 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:33.405038    2571 scope.go:117] "RemoveContainer" containerID="d234f18860391e97b06f16b909900c62d2fcd3764ceef061e87d11cc50dd60e1"
Jul 27 21:46:33.405233 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:46:33.405215    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d234f18860391e97b06f16b909900c62d2fcd3764ceef061e87d11cc50dd60e1\": container with ID starting with d234f18860391e97b06f16b909900c62d2fcd3764ceef061e87d11cc50dd60e1 not found: ID does not exist" containerID="d234f18860391e97b06f16b909900c62d2fcd3764ceef061e87d11cc50dd60e1"
Jul 27 21:46:33.405290 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:33.405242    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d234f18860391e97b06f16b909900c62d2fcd3764ceef061e87d11cc50dd60e1"} err="failed to get container status \"d234f18860391e97b06f16b909900c62d2fcd3764ceef061e87d11cc50dd60e1\": rpc error: code = NotFound desc = could not find container \"d234f18860391e97b06f16b909900c62d2fcd3764ceef061e87d11cc50dd60e1\": container with ID starting with d234f18860391e97b06f16b909900c62d2fcd3764ceef061e87d11cc50dd60e1 not found: ID does not exist"
Jul 27 21:46:33.409658 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:33.409636    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"]
Jul 27 21:46:33.415463 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:33.415445    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b2e96-predictor-7557b79bc4-bfcbn"]
Jul 27 21:46:34.571854 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:46:34.571820    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8df6bafa-cbe8-407a-a34c-e336c29ae291" path="/var/lib/kubelet/pods/8df6bafa-cbe8-407a-a34c-e336c29ae291/volumes"
Jul 27 21:52:50.209552 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:52:50.209516    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:53:43.353391 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:43.353360    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:53:53.568035 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:53.568005    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps"]
Jul 27 21:53:53.568429 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:53.568269    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps" podUID="42109737-e494-4909-8510-7683c70697c8" containerName="switch-graph-beb6a" containerID="cri-o://2366360e552fcbda141ba2371a543d2a61a184f13a539e6742475bdf8e5c44bd" gracePeriod=30
Jul 27 21:53:53.597864 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:53.597837    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:53:53.750432 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:53.750401    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:53:53.787503 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:53.787464    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"]
Jul 27 21:53:53.787699 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:53.787679    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm" podUID="240e7788-4b88-4b27-984f-0a35b90ceaa3" containerName="kserve-container" containerID="cri-o://455a16382717400dc47b20379bdaed21a66a651da434dd2150e3cce5cf3127b5" gracePeriod=30
Jul 27 21:53:53.787780 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:53.787726    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm" podUID="240e7788-4b88-4b27-984f-0a35b90ceaa3" containerName="kube-rbac-proxy" containerID="cri-o://d98087b257cd28fc129118ddd7471302abd689f588d3df91a489bfae9c79e70d" gracePeriod=30
Jul 27 21:53:53.813690 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:53.813670    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:53:54.452574 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:54.452545    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:53:54.513442 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:54.513413    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:53:54.865331 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:54.865296    2571 generic.go:356] "Generic (PLEG): container finished" podID="240e7788-4b88-4b27-984f-0a35b90ceaa3" containerID="d98087b257cd28fc129118ddd7471302abd689f588d3df91a489bfae9c79e70d" exitCode=2
Jul 27 21:53:54.865705 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:54.865356    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm" event={"ID":"240e7788-4b88-4b27-984f-0a35b90ceaa3","Type":"ContainerDied","Data":"d98087b257cd28fc129118ddd7471302abd689f588d3df91a489bfae9c79e70d"}
Jul 27 21:53:56.539703 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.539679    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"
Jul 27 21:53:56.579766 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.579726    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps" podUID="42109737-e494-4909-8510-7683c70697c8" containerName="switch-graph-beb6a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:53:56.617074 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.617046    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bvt5\" (UniqueName: \"kubernetes.io/projected/240e7788-4b88-4b27-984f-0a35b90ceaa3-kube-api-access-6bvt5\") pod \"240e7788-4b88-4b27-984f-0a35b90ceaa3\" (UID: \"240e7788-4b88-4b27-984f-0a35b90ceaa3\") "
Jul 27 21:53:56.617245 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.617092    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/240e7788-4b88-4b27-984f-0a35b90ceaa3-proxy-tls\") pod \"240e7788-4b88-4b27-984f-0a35b90ceaa3\" (UID: \"240e7788-4b88-4b27-984f-0a35b90ceaa3\") "
Jul 27 21:53:56.617245 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.617120    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/240e7788-4b88-4b27-984f-0a35b90ceaa3-kube-rbac-proxy-sar-config\") pod \"240e7788-4b88-4b27-984f-0a35b90ceaa3\" (UID: \"240e7788-4b88-4b27-984f-0a35b90ceaa3\") "
Jul 27 21:53:56.617537 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.617502    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/240e7788-4b88-4b27-984f-0a35b90ceaa3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "240e7788-4b88-4b27-984f-0a35b90ceaa3" (UID: "240e7788-4b88-4b27-984f-0a35b90ceaa3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:53:56.619360 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.619340    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/240e7788-4b88-4b27-984f-0a35b90ceaa3-kube-api-access-6bvt5" (OuterVolumeSpecName: "kube-api-access-6bvt5") pod "240e7788-4b88-4b27-984f-0a35b90ceaa3" (UID: "240e7788-4b88-4b27-984f-0a35b90ceaa3"). InnerVolumeSpecName "kube-api-access-6bvt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:53:56.619443 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.619354    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/240e7788-4b88-4b27-984f-0a35b90ceaa3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "240e7788-4b88-4b27-984f-0a35b90ceaa3" (UID: "240e7788-4b88-4b27-984f-0a35b90ceaa3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:53:56.717836 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.717755    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/240e7788-4b88-4b27-984f-0a35b90ceaa3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:53:56.717836 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.717782    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bvt5\" (UniqueName: \"kubernetes.io/projected/240e7788-4b88-4b27-984f-0a35b90ceaa3-kube-api-access-6bvt5\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:53:56.717836 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.717797    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/240e7788-4b88-4b27-984f-0a35b90ceaa3-proxy-tls\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:53:56.873386 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.873353    2571 generic.go:356] "Generic (PLEG): container finished" podID="240e7788-4b88-4b27-984f-0a35b90ceaa3" containerID="455a16382717400dc47b20379bdaed21a66a651da434dd2150e3cce5cf3127b5" exitCode=0
Jul 27 21:53:56.873547 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.873397    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"
Jul 27 21:53:56.873547 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.873409    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm" event={"ID":"240e7788-4b88-4b27-984f-0a35b90ceaa3","Type":"ContainerDied","Data":"455a16382717400dc47b20379bdaed21a66a651da434dd2150e3cce5cf3127b5"}
Jul 27 21:53:56.873547 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.873461    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm" event={"ID":"240e7788-4b88-4b27-984f-0a35b90ceaa3","Type":"ContainerDied","Data":"2a8a847dcce43def3d29f071aeb1669921e90f461a4d7ea0be000c3436079c16"}
Jul 27 21:53:56.873547 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.873487    2571 scope.go:117] "RemoveContainer" containerID="d98087b257cd28fc129118ddd7471302abd689f588d3df91a489bfae9c79e70d"
Jul 27 21:53:56.882279 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.882260    2571 scope.go:117] "RemoveContainer" containerID="455a16382717400dc47b20379bdaed21a66a651da434dd2150e3cce5cf3127b5"
Jul 27 21:53:56.889287 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.889272    2571 scope.go:117] "RemoveContainer" containerID="d98087b257cd28fc129118ddd7471302abd689f588d3df91a489bfae9c79e70d"
Jul 27 21:53:56.889516 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:53:56.889497    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d98087b257cd28fc129118ddd7471302abd689f588d3df91a489bfae9c79e70d\": container with ID starting with d98087b257cd28fc129118ddd7471302abd689f588d3df91a489bfae9c79e70d not found: ID does not exist" containerID="d98087b257cd28fc129118ddd7471302abd689f588d3df91a489bfae9c79e70d"
Jul 27 21:53:56.889590 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.889528    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d98087b257cd28fc129118ddd7471302abd689f588d3df91a489bfae9c79e70d"} err="failed to get container status \"d98087b257cd28fc129118ddd7471302abd689f588d3df91a489bfae9c79e70d\": rpc error: code = NotFound desc = could not find container \"d98087b257cd28fc129118ddd7471302abd689f588d3df91a489bfae9c79e70d\": container with ID starting with d98087b257cd28fc129118ddd7471302abd689f588d3df91a489bfae9c79e70d not found: ID does not exist"
Jul 27 21:53:56.889590 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.889552    2571 scope.go:117] "RemoveContainer" containerID="455a16382717400dc47b20379bdaed21a66a651da434dd2150e3cce5cf3127b5"
Jul 27 21:53:56.889779 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:53:56.889762    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"455a16382717400dc47b20379bdaed21a66a651da434dd2150e3cce5cf3127b5\": container with ID starting with 455a16382717400dc47b20379bdaed21a66a651da434dd2150e3cce5cf3127b5 not found: ID does not exist" containerID="455a16382717400dc47b20379bdaed21a66a651da434dd2150e3cce5cf3127b5"
Jul 27 21:53:56.889814 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.889785    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"455a16382717400dc47b20379bdaed21a66a651da434dd2150e3cce5cf3127b5"} err="failed to get container status \"455a16382717400dc47b20379bdaed21a66a651da434dd2150e3cce5cf3127b5\": rpc error: code = NotFound desc = could not find container \"455a16382717400dc47b20379bdaed21a66a651da434dd2150e3cce5cf3127b5\": container with ID starting with 455a16382717400dc47b20379bdaed21a66a651da434dd2150e3cce5cf3127b5 not found: ID does not exist"
Jul 27 21:53:56.893761 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.893741    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"]
Jul 27 21:53:56.897297 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:56.897277    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-beb6a-predictor-7c99f78746-rr2sm"]
Jul 27 21:53:58.571418 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:53:58.571385    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="240e7788-4b88-4b27-984f-0a35b90ceaa3" path="/var/lib/kubelet/pods/240e7788-4b88-4b27-984f-0a35b90ceaa3/volumes"
Jul 27 21:54:01.579820 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:01.579775    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps" podUID="42109737-e494-4909-8510-7683c70697c8" containerName="switch-graph-beb6a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:54:06.579257 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:06.579222    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps" podUID="42109737-e494-4909-8510-7683c70697c8" containerName="switch-graph-beb6a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:54:06.579628 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:06.579303    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps"
Jul 27 21:54:09.083259 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:09.083237    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb6a-78db6d8859-vxrps_42109737-e494-4909-8510-7683c70697c8/switch-graph-beb6a/0.log"
Jul 27 21:54:09.826357 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:09.826328    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb6a-78db6d8859-vxrps_42109737-e494-4909-8510-7683c70697c8/switch-graph-beb6a/0.log"
Jul 27 21:54:10.571352 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:10.571313    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb6a-78db6d8859-vxrps_42109737-e494-4909-8510-7683c70697c8/switch-graph-beb6a/0.log"
Jul 27 21:54:11.300291 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:11.300263    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb6a-78db6d8859-vxrps_42109737-e494-4909-8510-7683c70697c8/switch-graph-beb6a/0.log"
Jul 27 21:54:11.580107 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:11.580020    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps" podUID="42109737-e494-4909-8510-7683c70697c8" containerName="switch-graph-beb6a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:54:12.031600 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:12.031574    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb6a-78db6d8859-vxrps_42109737-e494-4909-8510-7683c70697c8/switch-graph-beb6a/0.log"
Jul 27 21:54:12.755758 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:12.755727    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb6a-78db6d8859-vxrps_42109737-e494-4909-8510-7683c70697c8/switch-graph-beb6a/0.log"
Jul 27 21:54:13.477556 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:13.477527    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb6a-78db6d8859-vxrps_42109737-e494-4909-8510-7683c70697c8/switch-graph-beb6a/0.log"
Jul 27 21:54:14.211193 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:14.211163    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb6a-78db6d8859-vxrps_42109737-e494-4909-8510-7683c70697c8/switch-graph-beb6a/0.log"
Jul 27 21:54:14.939796 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:14.939770    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb6a-78db6d8859-vxrps_42109737-e494-4909-8510-7683c70697c8/switch-graph-beb6a/0.log"
Jul 27 21:54:15.674820 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:15.674789    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb6a-78db6d8859-vxrps_42109737-e494-4909-8510-7683c70697c8/switch-graph-beb6a/0.log"
Jul 27 21:54:16.392845 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:16.392817    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb6a-78db6d8859-vxrps_42109737-e494-4909-8510-7683c70697c8/switch-graph-beb6a/0.log"
Jul 27 21:54:16.579512 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:16.579477    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps" podUID="42109737-e494-4909-8510-7683c70697c8" containerName="switch-graph-beb6a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:54:17.136731 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:17.136706    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb6a-78db6d8859-vxrps_42109737-e494-4909-8510-7683c70697c8/switch-graph-beb6a/0.log"
Jul 27 21:54:17.882911 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:17.882882    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb6a-78db6d8859-vxrps_42109737-e494-4909-8510-7683c70697c8/switch-graph-beb6a/0.log"
Jul 27 21:54:18.612794 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:18.612765    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-beb6a-78db6d8859-vxrps_42109737-e494-4909-8510-7683c70697c8/switch-graph-beb6a/0.log"
Jul 27 21:54:21.579176 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:21.579142    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps" podUID="42109737-e494-4909-8510-7683c70697c8" containerName="switch-graph-beb6a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:54:23.708876 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:23.708848    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps"
Jul 27 21:54:23.813036 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:23.813008    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42109737-e494-4909-8510-7683c70697c8-proxy-tls\") pod \"42109737-e494-4909-8510-7683c70697c8\" (UID: \"42109737-e494-4909-8510-7683c70697c8\") "
Jul 27 21:54:23.813227 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:23.813046    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42109737-e494-4909-8510-7683c70697c8-openshift-service-ca-bundle\") pod \"42109737-e494-4909-8510-7683c70697c8\" (UID: \"42109737-e494-4909-8510-7683c70697c8\") "
Jul 27 21:54:23.813460 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:23.813434    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42109737-e494-4909-8510-7683c70697c8-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "42109737-e494-4909-8510-7683c70697c8" (UID: "42109737-e494-4909-8510-7683c70697c8"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:54:23.815214 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:23.815194    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42109737-e494-4909-8510-7683c70697c8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "42109737-e494-4909-8510-7683c70697c8" (UID: "42109737-e494-4909-8510-7683c70697c8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:54:23.913984 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:23.913877    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42109737-e494-4909-8510-7683c70697c8-openshift-service-ca-bundle\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:54:23.913984 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:23.913905    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42109737-e494-4909-8510-7683c70697c8-proxy-tls\") on node \"ip-10-0-138-50.ec2.internal\" DevicePath \"\""
Jul 27 21:54:23.968309 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:23.968272    2571 generic.go:356] "Generic (PLEG): container finished" podID="42109737-e494-4909-8510-7683c70697c8" containerID="2366360e552fcbda141ba2371a543d2a61a184f13a539e6742475bdf8e5c44bd" exitCode=0
Jul 27 21:54:23.968309 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:23.968306    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps" event={"ID":"42109737-e494-4909-8510-7683c70697c8","Type":"ContainerDied","Data":"2366360e552fcbda141ba2371a543d2a61a184f13a539e6742475bdf8e5c44bd"}
Jul 27 21:54:23.968529 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:23.968322    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps"
Jul 27 21:54:23.968529 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:23.968365    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps" event={"ID":"42109737-e494-4909-8510-7683c70697c8","Type":"ContainerDied","Data":"63bad577293b406cc5fc9215e8451a66a95ec540a3581b12a2692e37a16acdfb"}
Jul 27 21:54:23.968529 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:23.968396    2571 scope.go:117] "RemoveContainer" containerID="2366360e552fcbda141ba2371a543d2a61a184f13a539e6742475bdf8e5c44bd"
Jul 27 21:54:23.976262 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:23.976245    2571 scope.go:117] "RemoveContainer" containerID="2366360e552fcbda141ba2371a543d2a61a184f13a539e6742475bdf8e5c44bd"
Jul 27 21:54:23.976510 ip-10-0-138-50 kubenswrapper[2571]: E0727 21:54:23.976494    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2366360e552fcbda141ba2371a543d2a61a184f13a539e6742475bdf8e5c44bd\": container with ID starting with 2366360e552fcbda141ba2371a543d2a61a184f13a539e6742475bdf8e5c44bd not found: ID does not exist" containerID="2366360e552fcbda141ba2371a543d2a61a184f13a539e6742475bdf8e5c44bd"
Jul 27 21:54:23.976556 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:23.976519    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2366360e552fcbda141ba2371a543d2a61a184f13a539e6742475bdf8e5c44bd"} err="failed to get container status \"2366360e552fcbda141ba2371a543d2a61a184f13a539e6742475bdf8e5c44bd\": rpc error: code = NotFound desc = could not find container \"2366360e552fcbda141ba2371a543d2a61a184f13a539e6742475bdf8e5c44bd\": container with ID starting with 2366360e552fcbda141ba2371a543d2a61a184f13a539e6742475bdf8e5c44bd not found: ID does not exist"
Jul 27 21:54:23.987939 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:23.987917    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps"]
Jul 27 21:54:23.989791 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:23.989771    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-beb6a-78db6d8859-vxrps"]
Jul 27 21:54:24.323083 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:24.323052    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:54:24.469066 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:24.469035    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:54:24.532906 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:24.532876    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:54:24.572296 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:24.572265    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42109737-e494-4909-8510-7683c70697c8" path="/var/lib/kubelet/pods/42109737-e494-4909-8510-7683c70697c8/volumes"
Jul 27 21:54:27.073341 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:27.073315    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kvpv2_71b80e11-ddee-4d86-a0b9-632732d0cca2/global-pull-secret-syncer/0.log"
Jul 27 21:54:27.206576 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:27.206550    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log"
Jul 27 21:54:27.360598 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:27.360527    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-50.ec2.internal_fda18b08661970af357d9091388de747/haproxy/0.log"
Jul 27 21:54:29.802406 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:29.802366    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:54:29.858234 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:29.858200    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5fg9l_70daf650-e6e8-4170-83e4-a288b081605e/konnectivity-agent/0.log" limit=1024
Jul 27 21:54:31.732819 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:31.732792    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p8swf_f1f4ab75-412a-48f7-a10b-440ee3bb02e4/node-exporter/0.log"
Jul 27 21:54:31.753080 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:31.753055    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p8swf_f1f4ab75-412a-48f7-a10b-440ee3bb02e4/kube-rbac-proxy/0.log"
Jul 27 21:54:31.776464 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:31.776442    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p8swf_f1f4ab75-412a-48f7-a10b-440ee3bb02e4/init-textfile/0.log"
Jul 27 21:54:34.967398 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:34.967366    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr"]
Jul 27 21:54:34.967775 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:34.967628    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="240e7788-4b88-4b27-984f-0a35b90ceaa3" containerName="kserve-container"
Jul 27 21:54:34.967775 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:34.967640    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="240e7788-4b88-4b27-984f-0a35b90ceaa3" containerName="kserve-container"
Jul 27 21:54:34.967775 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:34.967650    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8df6bafa-cbe8-407a-a34c-e336c29ae291" containerName="kube-rbac-proxy"
Jul 27 21:54:34.967775 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:34.967655    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8df6bafa-cbe8-407a-a34c-e336c29ae291" containerName="kube-rbac-proxy"
Jul 27 21:54:34.967775 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:34.967662    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8df6bafa-cbe8-407a-a34c-e336c29ae291" containerName="kserve-container"
Jul 27 21:54:34.967775 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:34.967669    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8df6bafa-cbe8-407a-a34c-e336c29ae291" containerName="kserve-container"
Jul 27 21:54:34.967775 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:34.967676    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="240e7788-4b88-4b27-984f-0a35b90ceaa3" containerName="kube-rbac-proxy"
Jul 27 21:54:34.967775 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:34.967681    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="240e7788-4b88-4b27-984f-0a35b90ceaa3" containerName="kube-rbac-proxy"
Jul 27 21:54:34.967775 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:34.967692    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42109737-e494-4909-8510-7683c70697c8" containerName="switch-graph-beb6a"
Jul 27 21:54:34.967775 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:34.967698    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="42109737-e494-4909-8510-7683c70697c8" containerName="switch-graph-beb6a"
Jul 27 21:54:34.967775 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:34.967734    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="240e7788-4b88-4b27-984f-0a35b90ceaa3" containerName="kserve-container"
Jul 27 21:54:34.967775 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:34.967741    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8df6bafa-cbe8-407a-a34c-e336c29ae291" containerName="kserve-container"
Jul 27 21:54:34.967775 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:34.967746    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8df6bafa-cbe8-407a-a34c-e336c29ae291" containerName="kube-rbac-proxy"
Jul 27 21:54:34.967775 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:34.967752    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="240e7788-4b88-4b27-984f-0a35b90ceaa3" containerName="kube-rbac-proxy"
Jul 27 21:54:34.967775 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:34.967757    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="42109737-e494-4909-8510-7683c70697c8" containerName="switch-graph-beb6a"
Jul 27 21:54:34.971918 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:34.971903    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr"
Jul 27 21:54:34.974439 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:34.974418    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6q2f4\"/\"kube-root-ca.crt\""
Jul 27 21:54:34.974567 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:34.974437    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6q2f4\"/\"openshift-service-ca.crt\""
Jul 27 21:54:34.974567 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:34.974482    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6q2f4\"/\"default-dockercfg-99gfb\""
Jul 27 21:54:34.980319 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:34.980299    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr"]
Jul 27 21:54:35.100991 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:35.100931    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/86256e3c-d694-4fec-932d-f2704f149db6-sys\") pod \"perf-node-gather-daemonset-qqwpr\" (UID: \"86256e3c-d694-4fec-932d-f2704f149db6\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr"
Jul 27 21:54:35.101168 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:35.101019    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/86256e3c-d694-4fec-932d-f2704f149db6-lib-modules\") pod \"perf-node-gather-daemonset-qqwpr\" (UID: \"86256e3c-d694-4fec-932d-f2704f149db6\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr"
Jul 27 21:54:35.101168 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:35.101044    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/86256e3c-d694-4fec-932d-f2704f149db6-proc\") pod \"perf-node-gather-daemonset-qqwpr\" (UID: \"86256e3c-d694-4fec-932d-f2704f149db6\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr"
Jul 27 21:54:35.101168 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:35.101058    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/86256e3c-d694-4fec-932d-f2704f149db6-podres\") pod \"perf-node-gather-daemonset-qqwpr\" (UID: \"86256e3c-d694-4fec-932d-f2704f149db6\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr"
Jul 27 21:54:35.101168 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:35.101072    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcczv\" (UniqueName: \"kubernetes.io/projected/86256e3c-d694-4fec-932d-f2704f149db6-kube-api-access-mcczv\") pod \"perf-node-gather-daemonset-qqwpr\" (UID: \"86256e3c-d694-4fec-932d-f2704f149db6\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr"
Jul 27 21:54:35.201713 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:35.201679    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/86256e3c-d694-4fec-932d-f2704f149db6-lib-modules\") pod \"perf-node-gather-daemonset-qqwpr\" (UID: \"86256e3c-d694-4fec-932d-f2704f149db6\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr"
Jul 27 21:54:35.201713 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:35.201717    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/86256e3c-d694-4fec-932d-f2704f149db6-proc\") pod \"perf-node-gather-daemonset-qqwpr\" (UID: \"86256e3c-d694-4fec-932d-f2704f149db6\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr"
Jul 27 21:54:35.202016 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:35.201735    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/86256e3c-d694-4fec-932d-f2704f149db6-podres\") pod \"perf-node-gather-daemonset-qqwpr\" (UID: \"86256e3c-d694-4fec-932d-f2704f149db6\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr"
Jul 27 21:54:35.202016 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:35.201753    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcczv\" (UniqueName: \"kubernetes.io/projected/86256e3c-d694-4fec-932d-f2704f149db6-kube-api-access-mcczv\") pod \"perf-node-gather-daemonset-qqwpr\" (UID: \"86256e3c-d694-4fec-932d-f2704f149db6\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr"
Jul 27 21:54:35.202016 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:35.201787    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/86256e3c-d694-4fec-932d-f2704f149db6-sys\") pod \"perf-node-gather-daemonset-qqwpr\" (UID: \"86256e3c-d694-4fec-932d-f2704f149db6\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr"
Jul 27 21:54:35.202016 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:35.201814    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/86256e3c-d694-4fec-932d-f2704f149db6-proc\") pod \"perf-node-gather-daemonset-qqwpr\" (UID: \"86256e3c-d694-4fec-932d-f2704f149db6\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr"
Jul 27 21:54:35.202016 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:35.201854    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/86256e3c-d694-4fec-932d-f2704f149db6-sys\") pod \"perf-node-gather-daemonset-qqwpr\" (UID: \"86256e3c-d694-4fec-932d-f2704f149db6\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr"
Jul 27 21:54:35.202016 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:35.201859    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/86256e3c-d694-4fec-932d-f2704f149db6-podres\") pod \"perf-node-gather-daemonset-qqwpr\" (UID: \"86256e3c-d694-4fec-932d-f2704f149db6\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr"
Jul 27 21:54:35.202016 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:35.201861    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/86256e3c-d694-4fec-932d-f2704f149db6-lib-modules\") pod \"perf-node-gather-daemonset-qqwpr\" (UID: \"86256e3c-d694-4fec-932d-f2704f149db6\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr"
Jul 27 21:54:35.210120 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:35.210100    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcczv\" (UniqueName: \"kubernetes.io/projected/86256e3c-d694-4fec-932d-f2704f149db6-kube-api-access-mcczv\") pod \"perf-node-gather-daemonset-qqwpr\" (UID: \"86256e3c-d694-4fec-932d-f2704f149db6\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr"
Jul 27 21:54:35.282176 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:35.282147    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr"
Jul 27 21:54:35.408408 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:35.408375    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr"]
Jul 27 21:54:35.411209 ip-10-0-138-50 kubenswrapper[2571]: W0727 21:54:35.411180    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod86256e3c_d694_4fec_932d_f2704f149db6.slice/crio-b3dc24238bfa0a96719304168461c3c1efed0ed3c1f8defdf9bf532a6b4f1c09 WatchSource:0}: Error finding container b3dc24238bfa0a96719304168461c3c1efed0ed3c1f8defdf9bf532a6b4f1c09: Status 404 returned error can't find the container with id b3dc24238bfa0a96719304168461c3c1efed0ed3c1f8defdf9bf532a6b4f1c09
Jul 27 21:54:36.000866 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:36.000830    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rc6n5_40a3772d-f99d-45ae-bb4e-3bd833d47da0/dns/0.log"
Jul 27 21:54:36.008916 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:36.008892    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr" event={"ID":"86256e3c-d694-4fec-932d-f2704f149db6","Type":"ContainerStarted","Data":"4a7ab755d87f5159e6a36fe5d5a8a308ed825e388752faa5205d462ba351f4d0"}
Jul 27 21:54:36.009071 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:36.008985    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr" event={"ID":"86256e3c-d694-4fec-932d-f2704f149db6","Type":"ContainerStarted","Data":"b3dc24238bfa0a96719304168461c3c1efed0ed3c1f8defdf9bf532a6b4f1c09"}
Jul 27 21:54:36.009071 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:36.009007    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr"
Jul 27 21:54:36.023903 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:36.023812    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr" podStartSLOduration=2.023793978 podStartE2EDuration="2.023793978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:54:36.023194256 +0000 UTC m=+3055.958410628" watchObservedRunningTime="2026-07-27 21:54:36.023793978 +0000 UTC m=+3055.959010358"
Jul 27 21:54:36.024070 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:36.023932    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rc6n5_40a3772d-f99d-45ae-bb4e-3bd833d47da0/kube-rbac-proxy/0.log"
Jul 27 21:54:36.108419 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:36.108380    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gtpdk_c9d12f6c-3f55-44b3-b1d5-be1215730520/dns-node-resolver/0.log"
Jul 27 21:54:36.567101 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:36.567067    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-89d6577f8-nz7f2_883d84e7-a560-4d61-92f9-7f98d8caead0/registry/0.log"
Jul 27 21:54:36.588641 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:36.588603    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cwvnf_3de75e46-ec68-4801-9493-b1b31f67eefb/node-ca/0.log"
Jul 27 21:54:37.706156 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:37.706128    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xpx7g_5d872ddb-8292-4ecc-b10b-98e6c884724e/serve-healthcheck-canary/0.log"
Jul 27 21:54:38.122395 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:38.122366    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fkz7m_98e950d5-0228-4c15-b5a3-8ad6a789afa8/kube-rbac-proxy/0.log"
Jul 27 21:54:38.141978 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:38.141937    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fkz7m_98e950d5-0228-4c15-b5a3-8ad6a789afa8/exporter/0.log"
Jul 27 21:54:38.162786 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:38.162759    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fkz7m_98e950d5-0228-4c15-b5a3-8ad6a789afa8/extractor/0.log"
Jul 27 21:54:42.022231 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:42.022204    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-qqwpr"
Jul 27 21:54:44.294714 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:44.294690    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-hm4x2_012cd20d-97a2-4d3d-804b-ee2f6302bdf2/migrator/0.log"
Jul 27 21:54:44.319398 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:44.319372    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-hm4x2_012cd20d-97a2-4d3d-804b-ee2f6302bdf2/graceful-termination/0.log"
Jul 27 21:54:45.762897 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:45.762824    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ttt2t_8d760a2e-48ae-4021-9766-7c875e7ec1c4/kube-multus-additional-cni-plugins/0.log"
Jul 27 21:54:45.784146 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:45.784121    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ttt2t_8d760a2e-48ae-4021-9766-7c875e7ec1c4/egress-router-binary-copy/0.log"
Jul 27 21:54:45.802604 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:45.802580    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ttt2t_8d760a2e-48ae-4021-9766-7c875e7ec1c4/cni-plugins/0.log"
Jul 27 21:54:45.824474 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:45.824451    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ttt2t_8d760a2e-48ae-4021-9766-7c875e7ec1c4/bond-cni-plugin/0.log"
Jul 27 21:54:45.847997 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:45.847972    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ttt2t_8d760a2e-48ae-4021-9766-7c875e7ec1c4/routeoverride-cni/0.log"
Jul 27 21:54:45.868362 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:45.868346    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ttt2t_8d760a2e-48ae-4021-9766-7c875e7ec1c4/whereabouts-cni-bincopy/0.log"
Jul 27 21:54:45.887456 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:45.887437    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ttt2t_8d760a2e-48ae-4021-9766-7c875e7ec1c4/whereabouts-cni/0.log"
Jul 27 21:54:45.983134 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:45.983108    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-x8x5c_72108aac-bf13-4304-8271-b32ad115d0ff/kube-multus/0.log"
Jul 27 21:54:46.094557 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:46.094526    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-whppm_222ea6ea-61b8-4854-bc88-834449ce2974/network-metrics-daemon/0.log"
Jul 27 21:54:46.112230 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:46.112209    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-whppm_222ea6ea-61b8-4854-bc88-834449ce2974/kube-rbac-proxy/0.log"
Jul 27 21:54:47.230917 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:47.230882    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xkcbq_e2b3b600-ca21-4067-aa49-92df5b7b4bfa/ovn-controller/0.log"
Jul 27 21:54:47.249763 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:47.249741    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xkcbq_e2b3b600-ca21-4067-aa49-92df5b7b4bfa/ovn-acl-logging/0.log"
Jul 27 21:54:47.272584 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:47.272558    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xkcbq_e2b3b600-ca21-4067-aa49-92df5b7b4bfa/kube-rbac-proxy-node/0.log"
Jul 27 21:54:47.294433 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:47.294388    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xkcbq_e2b3b600-ca21-4067-aa49-92df5b7b4bfa/kube-rbac-proxy-ovn-metrics/0.log"
Jul 27 21:54:47.314186 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:47.314163    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xkcbq_e2b3b600-ca21-4067-aa49-92df5b7b4bfa/northd/0.log"
Jul 27 21:54:47.333696 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:47.333678    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xkcbq_e2b3b600-ca21-4067-aa49-92df5b7b4bfa/nbdb/0.log"
Jul 27 21:54:47.353341 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:47.353311    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xkcbq_e2b3b600-ca21-4067-aa49-92df5b7b4bfa/sbdb/0.log"
Jul 27 21:54:47.400837 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:47.400809    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xkcbq_e2b3b600-ca21-4067-aa49-92df5b7b4bfa/ovnkube-controller/0.log"
Jul 27 21:54:48.397382 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:48.397353    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-m2bn8_c268e820-791d-448a-b5ec-83ec69ad08ee/network-check-target-container/0.log"
Jul 27 21:54:49.194033 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:49.194002    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-855bl_f2f1d470-36dd-429a-bdc9-191fa2e7b4df/iptables-alerter/0.log"
Jul 27 21:54:49.843692 ip-10-0-138-50 kubenswrapper[2571]: I0727 21:54:49.843660    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-fb28r_f1a6ba9d-9d7d-4df1-a3dc-71d063566c5f/tuned/0.log"